AUTHENTICATED 
U.S. GOVERNMENT 
INFORMATION ^ 


S. Hrg. 108-277 

KEEPING THE LIGHTS ON: THE FEDERAL ROLE 
IN MANAGING THE NATION’S ELECTRICITY 


HEARING 


BEFORE THE 

OVERSIGHT OF GOVERNMENT MANAGEMENT, 
THE FEDERAL WORKFORCE AND THE DISTRICT 
OF COLUMBIA SUBCOMMITTEE 

OF THE 

COMMITTEE ON 
GOVERNMENTAL AFFAIRS 
UNITED STATES SENATE 

ONE HUNDRED EIGHTH CONGRESS 

FIRST SESSION 


SEPTEMBER 10 AND NOVEMBER 20, 2003 


Printed for the use of the Committee on Governmental Affairs 



U.S. GOVERNMENT PRINTING OFFICE 
90-232 PDF WASHINGTON : 2004 


For sale by the Superintendent of Documents, U.S. Government Printing Office 
Internet: bookstore.gpo.gov Phone: toll free (866) 512—1800; DC area (202) 512-1800 
Fax: (202) 512-2250 Mail: Stop SSOP, Washington, DC 20402-0001 


COMMITTEE ON GOVERNMENTAL AFFAIRS 


SUSAN M. COLLINS, Maine, Chairman 

JOSEPH I. LTEBERMAN, Connecticut 
CARL LEVIN, Michigan 
DANIEL K. AKAKA, Hawaii 


TED STEVENS, Alaska 
GEORGE V. VOINOVICH, Ohio 
NORM COLEMAN, Minnesota 
ARLEN SPECTER, Pennsylvania 
ROBERT F. BENNETT, Utah 
PETER G. FITZGERALD, Illinois 
JOHN E. SUNUNU, New Hampshire 
RICHARD C. SHELBY, Alabama 


RICHARD J. DURBIN, Illinois 
THOMAS R. CARPER, Delaware 
MARK DAYTON, Minnesota 
FRANK LAUTENBERG, New Jersey 
MARK PRYOR, Arkansas 


Michael D. Bopp, Staff Director and Chief Counsel 
Joyce A. Rechtschaffen, Minority Staff Director and Counsel 
Amy B. Newhouse, Chief Clerk 


OVERSIGHT OF GOVERNMENT MANAGEMENT, THE FEDERAL 
WORKFORCE AND THE DISTRICT OF COLUMBIA SUBCOMMITTEE 


GEORGE V. VOINOVICH, Ohio, Chairman 


TED STEVENS, Alaska 
NORM COLEMAN, Minnesota 
ROBERT F. BENNETT, Utah 
PETER G. FITZGERALD, Illinois 
JOHN E. SUNUNU, New Hampshire 


RICHARD J. DURBIN, Illinois 
DANIEL K. AKAKA, Hawaii 
THOMAS R. CARPER, Delaware 
FRANK LAUTENBERG, New Jersey 
MARK PRYOR, Arkansas 


Andrew Richardson, Staff Director 
Marianne Clifford Upton, Minority Staff Director and Chief Counsel 
Cynthia Simmons, Chief Clerk 


(II) 



CONTENTS 


Opening statements: Page 

Senator Voinovich 1, 37 

Senator Levin 4 

Senator Lautenberg 7, 47 

WITNESSES 

Wednesday, September 10, 2003 

Kyle E. McSlarrow, Deputy Secretary of Energy, Department of Energy 8 

Pat Wood, III, Chairman, Federal Energy Regulatory Commission 10 

Alan R. Schriber, Chairman, Public Utility Commission of Ohio 20 

Craig A. Glazer, Vice President, PJM Interconnection, L.L.C 22 

James P. Torgerson, President and Chief Executive Officer, Midwest Inde- 
pendent Transmission System Operator, Inc 24 

William J. Museler, President and Chief Executive Officer; New York Inde- 
pendent System Operator 26 

James Y. Kerr, II, Commissioner, North Carolina Utilities Commission 28 

Mark N. Cooper, Director of Research, Consumer F ederation of America 30 

Thursday, November 20, 2003 

Pat Wood, III, Chairman, Federal Energy Regulatory Commission 40 

James W. Glotfelty, Director, Office of Electric Transmission and Distribution, 

U.S. Department of Energy 43 

Michael R. Gent, President and CEO, North American Electric Realiability 
Council 45 

Alphabetical List of Witnesses 

Cooper, Mark N.: 

Testimony 30 

Prepared statement 187 

Gent, Michael R.: 

Testimony 45 

Prepared statement with attachments 200 

Glazer, Craig A.: 

Testimony 22 

Prepared statement 98 

Glotfelty, James W.: 

Testimony 43 

Prepared statement 196 

Kerr, James Y. II: 

Testimony 28 

Prepared statement with attachments 127 

McSlarrow, Kyle E.: 

Testimony 8 

Prepared statement 61 

Museler, William J.: 

Testimony 26 

Prepared statement 118 

Schriber, Alan R.: 

Testimony 20 

Prepared statement 88 

Torgerson, James P.: 

Testimony 24 


(III) 



IV 


Page 


Torgerson, James P. — Continued 

Prepared statement Ill 

Wood, Pat III: 

Testimony 10, 40 

Prepared statements 71, 193 

Appendix 

“Interim Report: Causes of the August 14th Blackout in the United States 
and Canada,” by the U.S.-Canada Power System Outage Task Force, No- 
vember 2003 211 

“Reviving the Electricity Sector,” Findings of the National Commission on 

Energy Poilicy, August 2003 333 

“Contributing Factors to the Series of Outages on August 14, 2003,” A White 
Paper, September 23, 2003, by Stephen T. Lee, Ph.D., EPRI Project Man- 
ager 345 

“Michigan Public Service Commission Report on August 14th Blackout,” No- 
vember 2003 374 

Letter to Senator Domenici and Representative Tauzin, dated September 

2, 2003 484 

Responses to Questions for the Record for Pat Wood, III, from Senator Lau- 
tenberg 486 



KEEPING THE LIGHTS ON: THE FEDERAL 
ROLE IN MANAGING THE NATION’S ELEC- 
TRICITY 


WEDNESDAY, SEPTEMBER 10, 2003 

U.S. Senate, 

Oversight of Government Management, the Federal 

Workforce, and the District of Columbia Subcommittee, 
of the Committee on Governmental Affairs, 

Washington, DC. 

The Subcommittee met, pursuant to notice, at 9:02 a.m., in room 
SD-342, Dirksen Senate Office Building, Hon. George V. Voinovich, 
Chairman of the Subcommittee, presiding. 

Present: Senators Voinovich, Levin, and Lautenberg. 

OPENING STATEMENT OF SENATOR VOINOVICH 

Senator Voinovich. The Subcommittee will please come to order. 
Because we have a very busy schedule and many witnesses, I want 
to begin this hearing on time. I am sure that my colleagues will 
be coming, as I expect several of them to be here today. They indi- 
cated that they will be here. 

Again, thank you for coming. We are here today to discuss the 
Federal role in ensuring the reliability of our electricity supply. In 
order to ensure that we have reliability, we need to have highly de- 
pendent systems to operate continuously and for them to generate 
and transmit electricity. It is axiomatic that adequate generation 
is meaningless without the transmission capacity to deliver elec- 
tricity, and equally so to have an adequate or even robust trans- 
mission system without adequate generation to meet customer de- 
mand. 

There is no question that this Nation is currently served by a 
strained electricity system. Generation has failed to meet growing 
demand and facing ever-tightening restrictions that limit our abil- 
ity to expand generation capacity, transmission capacity, and 
transmission capacity increases have lagged even behind genera- 
tion increases, let alone demand increases. So, in effect, what we 
have had is a lot more generation, but we haven’t had the trans- 
mission capacity to keep up with that generation capacity. 

Fortunately, the Senate, along with the House and President 
Bush, is moving forward in developing an energy policy to help al- 
leviate these constraints. The energy bill will encourage increased 
production and supplies of natural gas and expansion of hydro- 
based nuclear and clean coal-fired generation. 

(l) 
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Further, I have introduced President Bush’s Clear Skies Act, 
which will provide much needed relief to natural gas markets by 
protecting the long-term viability of coal-based generation and pro- 
vide regulatory certainty for utilities, which is something that we 
need desperately in this country. There is so much uncertainty 
today out there among generators that it has never been like this 
in the country’s history. 

The issue of electric reliability was brought front and center on 
August 14, when more than 50 million Americans and Canadians 
lost power in parts of the Northeast, the Midwest, and Ontario. In 
my home State of Ohio, at least two million — two million people 
were affected, including yours truly. It was very nice that day 
when the lights came on about 12 hours exactly from the time that 
they went out. I landed at Hopkins Airport when the lights went 
out and it took us about 2 hours to go through the whole procedure. 
I give the bag handlers a lot of credit for doing everything by hand. 

First, I would like to commend the administration for its leader- 
ship. President Bush moved quickly to create the U.S. -Canada 
Joint Task Force on the power outage. By initiating a thorough ex- 
amination of the blackout and getting the Canadians involved 
quickly, the administration has put us on a path to discover what 
happened. 

The purpose of today’s hearing is not to focus on who was respon- 
sible for the blackout. I want to emphasize that. I am confident 
that all of those questions will be answered by the investigation 
being conducted by the task force, and in the future, we will be 
holding hearings on these findings. Rather, today’s purpose is to 
focus on how we prevent something similar from happening again. 

Many of us have known for several years that power trans- 
mission has not kept up with power generation, particularly in the 
era of deregulation, where the power grid takes on much greater 
priority. The point is, even if the power hadn’t gone out, we would 
still be facing potential problems with our outdated and inadequate 
transmission system. 

What action must be taken by the Federal Government to ensure 
that our grid doesn’t collapse again in the future? The August 
blackout was only the most recent example of why our country’s 
electricity transmission system needs to be modernized. Like all of 
my colleagues on this Subcommittee and in this Senate, I am truly 
concerned about our Nation’s energy situation. 

It may surprise many people in this room that Ohio is the third 
largest energy-consuming State in the United States of America, 
behind California and New York. Ohio, as well as other States, 
needs affordable, reliable energy to ensure a robust economy and 
future growth. Unfortunately, this blackout has had a costly effect 
on many of our business interests and sectors. The Ohio Manufac- 
turers’ Association estimates that the August blackout directly cost 
Ohio manufacturers over $1 billion — $1 billion in an economy that 
is sputtering. 

For example, Republic Engineered Products, based in Ohio, expe- 
rienced a fire and an explosion as a result of the blackout which 
seriously damaged one of their blast furnaces in Lorraine, Ohio. 
After this furnace went offline, the most effective solution for the 
company was to start a back-up furnace, which cost a significant 
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amount of money. Additionally, there was molten steel in their 
process machinery which cooled when power was lost, causing addi- 
tional damage to their equipment. The final price tag this company 
is facing because of the blackouts will be in the millions of dollars. 
I have heard similar stories from companies in other States af- 
fected by power outage, including New York, Pennsylvania, and 
Michigan. 

Over the last 40 years, our national electricity system has be- 
come congested and strained because growth in electricity demand 
has not been matched by corresponding investment in new genera- 
tion and transmission. Further, our existing transmission infra- 
structure, which was created in a world where utilities operated as 
virtual monopolies within their geographic regions, was not de- 
signed to meet the demands of our modern electricity markets. 

The result is that generation shortages and transmission con- 
straints have led to major blackouts in California — we have forgot- 
ten that so quickly, but it happened — the Midwest, and the North- 
east, and the risks of further blackouts are growing daily. Many of 
us were concerned last summer that New York would experience 
blackouts because of the lack of transmission capacity in that great 
city. 

This is not acceptable. When my constituents flip on the lights, 
they expect that the lights come on. 

I am pleased that modernization of our transmission systems is 
finally a national priority and is being debated in the energy bill 
Conference Committee. The need for reliable transmission is uni- 
versally recognized. However, things break down when policy mak- 
ers argue about who pays for it. We went through that last year 
when we were trying to put the electricity title together for the en- 
ergy bill. Who pays for it? Is it the utilities? Is it the customers? 
There is an argument. Somehow, that has got to be worked out so 
that it is a fair way of paying for that transmission capacity. 

The fact is that investing in power generation is a better invest- 
ment than investing in transmission, and somehow, we have to rec- 
ognize that. It is understandable that companies are hesitant to in- 
vest in transmission because they are facing obstacles, such as 
NIMBY, “not in my backyard,” and frankly, I have heard, and I am 
sure Senator Levin has heard from constituents that don’t want 
transmission lines through their farm and through wherever they 
live. 

To make matters worse, it seems there is a new movement afoot. 
It is called the BANANA movement, “build absolutely nothing any- 
where near anything.” Further, even companies that are willing to 
invest in transmission are reluctant because of burdensome and on- 
erous environmental requirements. 

We have to harmonize our environmental and energy policies to 
keep this Nation competitive. For too long, our environmental con- 
cerns, our needs, and our energy concerns have been going in dif- 
ferent directions. They haven’t been speaking to each other. For the 
betterment of this country, we need to harmonize those environ- 
mental and energy needs or we are in deep, deep trouble. 

In an effort to increase transmission capacity, I sponsored legis- 
lation in the last Congress to amend the National Environmental 
Policy Act and streamline the siting process for transmission cor- 
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ridors. Further, I have recently sent a letter to the energy bill con- 
ferees encouraging them to include provisions to strengthen the re- 
liability of our Nation’s electricity supply in that energy bill. 1 

What I would like to hear from each witness today is your 
thoughts on the best way to move forward to modernize the grid 
and what is the appropriate Federal role in managing and regu- 
lating the grid. 

I now would like to call on Senator Levin. Senator Levin, I am 
very happy that you are here today, and if you have an opening 
statement, we would like to hear it. 

OPENING STATEMENT OF SENATOR LEVIN 

Senator Levin. Thank you, and I will ask that my entire state- 
ment be made part of the record. I will try to be brief because I 
know that we have a number of votes coming up, I believe a little 
later on this morning. 

I commend you, Mr. Chairman, for not only calling this hearing, 
for your initiative and for your leadership in this area, but also for 
trying to hold a hearing when we have a series of stacked votes 
going on later on this morning. It is going to be quite a challenge 
to do that and I thank you for that effort. 

Over the past few years, our country and our economy have been 
rocked by two major energy crises. The first, triggered by Enron’s 
collapse, disclosed rampant energy price manipulation and fraud 
and billions of dollars in electricity overcharges in a number of 
States. The second, on August 14, was a sudden and massive power 
outage that disrupted broad sections of the Midwest, New York, 
and Canada. 

In my home State of Michigan alone, over six million residents 
lost power. Numerous Michigan businesses and schools closed, in- 
cluding over 70 manufacturing companies. The City of Detroit and 
much of Southeast Michigan lost the ability to operate water and 
sewer systems. Michigan State and local governments spent more 
than $20 million on emergency assistance. And ongoing assess- 
ments of losses associated with the power failure in Michigan are 
expected to reach $1 billion. 

The massive power failure of August 2003 on top of the massive 
price manipulation perpetrated by Enron and others provide addi- 
tional proof, if more were needed, that the deregulated energy mar- 
kets in this country are not functioning well. These energy markets 
are not self-policing. There is no invisible hand guaranteeing effi- 
cient power flows and fair prices. Instead, a philosophy that U.S. 
energy markets can function free of meaningful oversight has 
thrown our energy markets into turmoil and opened the door to 
power failures and price manipulation, punishing U.S. ratepayers 
and taxpayers with economic disruption and high energy costs. 

The result is not only unreliable power that threatens our econ- 
omy and our security, but also a loss of investor confidence in U.S. 
energy markets, a dearth of new investment, and the bankruptcy 
of some energy companies once at the heart of our U.S. energy in- 
dustry. 


ir The letter to Senator Domenici and Representative Tauzin, dated September 3, 2003, ap- 
pears in the Appendix on page 484. 
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Despite the growing complexity and difficulties associated with 
power transmission, it is also clear that currently no single agency 
or company can be held accountable for ensuring the reliability of 
the Nation’s electric grid. Control of transmission lines varies 
across the Nation and includes a hodgepodge of Federal, regional, 
State, local, and private agencies and entities. 

One key player testifying today, the Midwest Independent Sys- 
tem Operator, MISO, schedules wholesale use of lines to transmit 
electricity but claims to lack the authority to interrupt power 
transmissions traveling from energy generators to consumers. In- 
stead, when lines go down, as they did on August 14, the MISO can 
issue instructions to utilities using the power lines, but must rely 
on voluntary compliance by the utilities to resolve fast-moving grid 
problems. 

The North American Electric Reliability Council, an industry 
group that designs and issues standards to ensure grid reliability, 
is dependent on voluntary compliance and cannot require trans- 
mission line users to meet its standards or penalize non-compli- 
ance. 

The Federal Energy Regulatory Commission (FERC) also says it 
lacks clear authority to ensure grid reliability, although in 1999 it 
issued rules which, in part, required regional transmission organi- 
zations to ensure reliability. 

The August power failure and the Enron price manipulation 
scandal provide clear mandates for Congressional action. Congress 
needs to pass legislation this year to increase grid reliability and 
to stop energy price manipulation. 

With respect to reliability, Congress needs to replace voluntary 
reliability standards with mandatory and enforceable reliability 
rules applicable to all users, owners, and operators of the trans- 
mission network. While industry-developed standards provide a 
starting point, the responsibility for writing the final rules to en- 
sure grid reliability needs to be vested in FERC, a Federal agency 
that can be held accountable for problems. 

Now, while some in Congress want to include a reliability provi- 
sion in the larger energy bill now in conference, using the urgency 
of that issue to propel enactment of the whole energy bill, this 
issue is too important to hold the reliability provision hostage to 
resolution of a myriad of other problems in the energy bill. We 
ought to act on the consensus that now exists to resolve the reli- 
ability problem. If we fail to act now, we are risking more black- 
outs. 

And just one word on this other legislation to eliminate the so- 
called Enron exemption, Mr. Chairman. I will not go into that in 
detail except to say that Senators Feinstein, Lugar, myself, and a 
number of other Senators have been working on anti-fraud and 
anti-manipulation laws. We have been guaranteed and assured 
that we will have an up-down vote on an amendment to the agri- 
culture appropriations bill to address that issue. We will be releas- 
ing a copy of the amendment in the near future and I, again, would 
appreciate the entire statement on that subject and other parts be 
made part of the record. 

Senator Voinovich. Thank you, Senator Levin. 

Senator Levin. Thank you very much, Mr. Chairman. 



6 


[The prepared opening statement of Senator Levin follows:] 

PREPARED OPENING STATEMENT BY SENATOR LEVIN 

Over the past few years, our country and our economy have been rocked by two 
major energy crises. The first, triggered by Enron’s collapse, disclosed rampant en- 
ergy price manipulation and fraud and billions of dollars in electricity overcharges 
in a number of states. The second, on August 14, was a sudden and massive power 
outage that disrupted broad sections of the Midwest, New York and Canada. In my 
home state, alone, over 6 million Michigan residents lost power. Numerous Michi- 
gan businesses and schools closed, including over 70 manufacturing companies. The 
City of Detroit and much of southeast Michigan lost the ability to operate water and 
sewer systems. Michigan state and local governments spent more than $20 million 
on emergency assistance, and ongoing assessments of losses associated with the 
power failure in Michigan are expected to reach $1 billion. 

The massive power failure of August 2003, on top of the massive price manipula- 
tion perpetrated by Enron and others, provide additional proof, if more were needed, 
that the United States’ deregulated energy markets are not functioning well. These 
energy markets are not self-policing — there is no invisible hand guaranteeing effi- 
cient power flows and fair prices. Instead, an ill-advised philosophy that U.S. energy 
markets can function free of meaningful standards and oversight has opened the 
door to power failures and price manipulation, punishing U.S. ratepayers and tax- 
payers with economic disruption and high energy costs. 

The result is not only unreliable power that threatens our economy and security, 
but also a loss of investor confidence in U.S. energy markets, a dearth of new invest- 
ment, and the bankruptcy of some energy companies once at the heart of the U.S. 
energy industry. 

Reforms can and must tackle these issues. While the precise causes of the August 
power failure have yet to be determined, and investigations by the Michigan Public 
Service Commission, U.S. Department of Energy in conjunction with Canada, and 
others are ongoing, some facts have already become clear. We know, for example 
that, during the 1990’s, utilities in the Northeast and Midwest underwent extensive 
deregulation that separated electricity power generation from power transmission 
over the grid. Since this deregulation, according to a recent report by the North 
American Electric Reliability Council, the Midwest has become one of the great 
crossroads in the transmission of power across the nation. Power produced as far 
away as Denver flows through the Midwestern grid on its way to users in New York 
and elsewhere. 

At the same time, the complexities and difficulties involved in coordinating power 
flowing through Midwestern transmission lines have increased. A patchwork of com- 
panies and utilities generating power use these lines. These transmission line users 
apparently have no legal obligation to alert the transmission line operators to up- 
coming voltage fluctuations or the cause of these fluctuations, even though power 
surges can overwhelm transmission lines with little notice and devastating impact. 
Utilities have no legal obligation to warn neighboring utilities of transmission prob- 
lems, even though warnings can play a crucial role in activating safeguards to mini- 
mize grid problems. Detroit Edison reported, for example, that on August 14, it had 
no idea there were problems on the grid until 2 minutes before the Michigan power 
loss, when power flowing from Michigan to Ohio jumped by 2,000 megawatts in 10 
second sand, 90 seconds later, power flowing from Ontario to Michigan jumped by 
2,600 megawatts. Thirty-seconds after that, Detroit Edison’s portion of the grid was 
dead. 

Despite the growing complexity and difficulties associated with power trans- 
mission, it is also clear that, currently, no single agency or company can be held 
accountable for ensuring the reliability of the nation’s electrical grid. Control of 
transmission lines varies across the nation and includes a hodge-podge of federal, 
regional, state, local and private agencies and entities. One key player testifying 
today, the Midwest Independent System Operator (MISO), schedules wholesale use 
of liens to transmit electricity, but claims to lack the authority to interrupt power 
transmissions traveling from generators to consumers. Instead, when lines go down 
as they did on August 14, the MISO says that it can issue instructions to utilities 
using the power lines, but must rely on voluntary compliance by the utilities to re- 
solve fast-moving grid problems. The North American Electric Reliability Council, 
an industry group that designs and issues standards to ensure grid reliability, is 
dependent on voluntary compliance and cannot require transmission line users to 
meet its standards or penalize noncompliance. The Federal Energy Regulatory Com- 
mission also says it lacks clear authority to ensure grid reliability although, in 1999, 
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it issued rules which, in part, require Regional Transmission Organizations to main- 
tain short-term reliability in their grid operations. 

The August power failure and the Enron price manipulation scandal provide clear 
mandates for Congressional action. Congress needs to pass legislation this year to 
increase grid reliability and stop energy price manipulation. With respect to reli- 
ability, Congress needs to replace voluntary reliability standards with mandatory 
and enforceable reliability rules applicable to all users, owners, and operators of the 
transmission network. While industry-developed standards provide a starting point, 
the responsibility for writing the final rules to ensure grid reliability needs to be 
vested in FERC, a federal agency that can be held accountable for problems. While 
some in Congress want to include reliability requirements in the larger energy bill 
now in conference, using the urgency of this issue to propel enactment of the whole 
bill, this issue is too important to hold resolving the reliability problem hostage to 
the resolution of a myriad of other problems in the energy bill. We ought to act now 
on the consensus that has apparently arisen on the need to strengthen grid reli- 
ability. If we fail to act now, we are risking more blackouts. 

Congress also needs to enact legislation to eliminate the so-called Enron exemp- 
tion that, in 2000, exempted certain energy transactions from federal oversight and 
federal anti-fraud and anti-manipulation laws. Senators Feinstein, Lugar, and I, as 
well as a number of other Senators, have been working on this legislation. When 
the Senate simply adopted last year’s energy bill without addressing this issue, the 
Senate Majority Leader promised us an up-or-down vote on an amendment to the 
Agriculture appropriations bill this fall, and we plan to circulate a bipartisan 
amendment addressing this issue in the near future. It is incomprehensible to me 
that some people are lobbying to maintain the Enron exemption and to stop efforts 
to strengthen federal anti-fraud and anti-manipulation laws. It is long past time for 
Congress to make it clear that fraud and price manipulation are unacceptable in 
any sector of the U.S. energy markets. 

I commend the Chairman for holding this hearing and look forward to the testi- 
mony. 

Senator Voinovich. Senator Lautenberg. 

OPENING STATEMENT OF SENATOR LAUTENBERG 

Senator Lautenberg. Thanks, Mr. Chairman. I am appreciative 
that you are calling today’s hearing on the blackout. It is so fresh 
in our memory and so dark a spot on the horizon. 

The loss of power on August 14 brought a large swathe of the 
country to a standstill and reportedly cost New York City alone $1 
billion. Now, in my opinion, this event dramatically demonstrates 
where we are headed if we fail to modernize the Nation’s electrical 
transmission system, but other factors may also have been involved 
and we will need answers about exactly what happened and why 
it happened. If we fail to correct the flaws in these systems and 
soon, many believe more regions will be brought to their knees. 

Shutdowns come with a high price tag: Massive public inconven- 
ience; increased dangers for Americans, especially for Americans 
like those who are in the air depending on air traffic controllers 
with their electric systems to safely land them. 

But reliable electricity is not a new issue. Since 1978, the coun- 
try has been inching toward deregulation and some regions have 
made great progress while others locked in outmoded systems dat- 
ing back to the beginning of electricity regulation in this country. 
I understand that some of my colleagues have concerns about elec- 
tric industry deregulation. But in my State of New Jersey, we are 
part of the PJM interconnection. It is the country’s first fully oper- 
ating regional transmission organization, the world’s largest com- 
petitive wholesale electricity market. 

The ratepayers in New Jersey have reaped enormous benefits 
from belonging to this RTO, including stable rates that have not 
risen in 11 years. Just as important, and more to the point of this 
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hearing, during the blackout, only 7 percent of PJM’s 25 million 
customers lost their power. Now, 7 percent is not insignificant, but 
compared to 98 or 100 percent, it is pretty darn good. 

So I would like to hear from today’s witnesses about their views 
on additional RTOs that centrally dispatch information, like PJM. 
Had more of these been operating on August 14, the question is, 
could the blackout have been better contained? 

The Wall Street Journal noted in an August 18 editorial that 
adequately addressing the issue of electricity reliability will take 
political will and regulatory common sense, and it is my hope that 
my fellow Senators and I will rise above the entrenched political 
positions that are not serving the public’s needs for a modern, reli- 
able electricity grid. Today, I am looking forward to hearing some 
regulatory common sense from these witnesses. 

I thank you, Mr. Chairman. I hope we will arrive at some direc- 
tion to take to cure this problem. 

Senator Voinovich. Thank you, Senator. 

We have an impressive lineup of witnesses this morning and I 
look forward to an informative discussion. As Senator Levin said, 
we have stacked votes, so this is going to be interesting to see how 
this all works out. 

Our first witness today is Deputy Secretary of Energy Kyle 
McSlarrow. He is joined by Patrick Wood, the Chairman of the 
Federal Energy Regulatory Commission. I thank you both for testi- 
fying today. I would also like to note that Jimmy Glotfelty, the Di- 
rector of the Office of Transmission and Distribution, is also at- 
tending this hearing. 

As is the custom here, I would like you to please rise and be 
sworn in. 

Do you swear the testimony that you are about to give before the 
Subcommittee is the truth, the whole truth, and nothing but the 
truth, so help you, God? 

Mr. McSlarrow. I do. 

Mr. Wood. I do. 

Senator Voinovich. The record will reflect that they answered in 
the affirmative. 

Mr. McSlarrow, we will start with you. 

TESTIMONY OF KYLE E. McSLARROW , 1 DEPUTY SECRETARY 
OF ENERGY, DEPARTMENT OF ENERGY 

Mr. McSlarrow. Mr. Chairman, thank you. Let me first say that 
we appreciate your leadership on such important issues as trans- 
mission reliability and a robust, competitive wholesale market that 
benefits consumers. I would like to note that you and Senator 
Levin were among the first to contact our Department to ensure 
that we were fully engaged when the blackout occurred on August 
14. Since then, as you mentioned, we have made good progress in 
our effort to determine the causes. 

As you know, within a few hours of last month’s blackout, Presi- 
dent Bush and Prime Minister Chretien ordered a cooperative, bi- 
national investigation into that incident. Top government officials 
from both countries and scores of technical and engineering experts 


lr The prepared statement of Mr. McSlarrow appears in the Appendix on page 61. 
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have been hard at work ever since to determine exactly what 
caused this outage, how it was allowed to spread, and what can be 
done to reduce the chances of such an incident in the future. 

While Secretary Abraham has the lead on this task force, I do 
want to recognize the personal leadership exhibited by Chairman 
Wood and all of FERC staff, who put in long hours to ensure that 
we get the right answers as quickly as possible. 

Once we have determined the causes of the blackout, we will 
enter Phase II of the task force’s two-part assignment, which is for- 
mulating recommendations to address the specific problems we un- 
cover. Any recommendations the joint U.S. -Canada task force 
makes will likely focus on technical standards for operation and 
maintenance of the grid and on the management of performance of 
the grid in order to more quickly correct the problems we identify. 
We are determined to complete this inquiry in a timely manner. 
We hope to have conclusions and recommendations in a matter of 
weeks, not months. As Secretary Abraham has said, we will not 
compromise quality for speed. 

Beyond the investigation, there is also the broader focus on the 
Federal role in electricity reliability. The President’s National En- 
ergy Policy noted that one of the greatest energy challenges was to 
improve our Nation’s aging energy infrastructure and particularly 
the transmission infrastructure. The National Energy Policy also 
called for a transmission grid study, which our Department con- 
ducted and completed in May 2002. That study outlined the cur- 
rent condition of our grid and recommended ways to promote the 
expansion of overall transmission capacity, elimination of the bot- 
tlenecks on the grid, and enhancement of the grid’s technical effi- 
ciency and improvement of the system’s reliability. 

There are several measures before the House and Senate Con- 
ference on the energy bill right now that would codify some of the 
recommendations included in the President’s National Energy Plan 
and our Department’s grid study. Since the President’s first days 
in office, the administration has strongly supported proposals to es- 
tablish mandatory and enforceable reliability standards to reduce 
the risk of power outages. We were pleased that both the House 
and the Senate included provisions in the energy bill to establish 
those standards. 

We also support proposals that would expand investment in 
transmission and generation facilities by repealing the Public Util- 
ity Holding Company Act, which has limited the resources that can 
be invested in a transmission system. 

We strongly support measures to provide greater regulatory cer- 
tainty for transmission expansion, including provisions providing 
for last resort Federal siting authority for high-priority trans- 
mission lines and providing for the coordination and streamlining 
of transmission permitting activities across Federal lands. 

We support options that would allow for increased rates of return 
on new transmission investments, including clarifying FERC’s au- 
thority to provide incentive-based rates to promote capital invest- 
ment in new transmission. 

We support the goal of regional coordination and planning 
through the mechanism of voluntary regional transmission organi- 
zations that would provide certainty to the marketplace, prevent 



10 


undue discrimination, and assist in eliminating transmission con- 
straints. And we support changes in Federal tax law to allow the 
recognition of gain over 8 years for the sale or disposition of trans- 
mission assets as part of restructuring and to allow rural coopera- 
tives to provide open access to their transmission systems without 
losing their tax-exempt status. 

Before I close, Mr. Chairman, let me just add that government 
research and development has an important role to play here, as 
well. That is why the President’s 2004 budget request includes ad- 
ditional funding for high-capacity technologies, such as high tem- 
perature superconducting transmission lines and for real-time grid 
management tools to enhance reliability. 

Our electricity system is the backbone of the U.S. economy. We 
probably don’t think about something so obvious until the lights go 
out, but we cannot afford to let such a vital component of our infra- 
structure fail and I am confident that Congress will send an energy 
bill to the President this fall that sets us on a course to successfully 
address those challenges. 

Mr. Chairman, that concludes my testimony. 

Senator Voinovich. Thank you very much. Chairman Wood. 

TESTIMONY OF PAT WOOD, III , 1 CHAIRMAN, FEDERAL ENERGY 
REGULATORY COMMISSION 

Mr. Wood. Thank you, Mr. Chairman, Senator Levin, and Sen- 
ator Lautenberg. As the Deputy Secretary just outlined, the FERC 
and its staff are participating as members of the joint U.S.-Canada 
Task Force to look at what actually happened, why it cascaded 
across such a broad territory. And while the analysis of this voltage 
collapse is ongoing, I will refrain from, as you indicate, for the pur- 
pose of this hearing, we are not going into that anyway, but will 
refrain from trying to analyze too deeply until we get the facts, be- 
cause I think the facts should drive us wherever they lead. 

The Federal role in electricity grid management is the focus of 
your hearing today, and from the outset, let me be clear. As I be- 
lieve Senator Levin just pointed out, our explicit authorities under 
the Federal Power Act and other statutes in the area of reliability 
are very limited. However, we do have some insight into regional 
grid management deriving from our role primarily as the economic 
regulator of the electricity industry and I think that can provide 
some insight, as you requested in your letter of invitation, Mr. 
Chairman. 

The blackout illustrates, as have many other events in the past 
couple of years, that the power grids are regional in nature. After 
each significant blackout, which were 1996, 1999, 2000, 2001 in 
California, and this year 2003, the Nation has taken significant 
steps forward to recognize that we are, in fact, interconnected and 
regional and to develop new rules and institutions that recognize 
this fact. 

For example, after the 1965 blackout in the Northeast, the 
NERC was created to create regional and voluntary reliability 
standards. That is the subject of the legislation in conference today. 
That would actually make that mandatory. 


lr rhe prepared statement of Mr. Wood appears in the Appendix on page 71. 
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After the 1996 blackout across the West, the governors and insti- 
tutions out there developed a regional transmission plan that led 
to significant investment in the grid. I do have to add that I think 
that needs to continue. I believe that this blackout, similarly, will 
warrant significant action toward better regional grid management, 
as well. 

Prior to my term on the Commission, the bipartisan commission 
back in the 1990’s issued Order 2000, which encouraged, but did 
not require, the formation of regional transmission organizations, 
such as PJM that Mr. Lautenberg talked about. The expectation at 
the time was that these would form across the country by the end 
of 2001 and that they would be able to assume full control of the 
regional grids. 

The Midwest, however, is a good example of the fits and starts 
and the difficulties that the voluntary RTO program has had in 
achieving independent regional control in many areas. In such a 
voluntary arrangement, key principles like governance, independ- 
ence, and even reliability are subject to negotiation and com- 
promise. 

In Order 2000, the Commission stated that RTOs would have 
public accountability for reliability. RTOs would improve reliability 
because they have a broader, more regional perspective than indi- 
vidual local utilities, which number presently 130 independent con- 
trol areas in the country that are actually the managers of the grid 
implementing the voluntary NERC standards. 

RTOs, as we have seen, have the ability to take action if they 
have the operational authority to do so. An RTO can end the bal- 
kanization within the region of the grid and assign it exclusive au- 
thority for short-term reliability. In fact, these characteristics are 
embedded in Order 2000 and they relate to the appropriate inde- 
pendence of the entity to run the grid, the appropriate scope and 
regional size of the grid configuration, full operational authority of 
the grid, and the exclusive authority to oversee short-term reli- 
ability, which are the NERC standards. 

RTOs also provide long-term transmission infrastructure devel- 
opment, a regional planning process, a regional rate recovery proc- 
ess through tariffs that are administered and are applied on a 
multi-state basis. These are developed through stakeholder proc- 
esses and have been very successful in the regions where we have 
them. I think that regional investments are going to be almost im- 
possible to make if you do not provide investors a very clear path 
for recovery of those new investments that I think, Mr. Chairman, 
you have pointed out. As a common sense matter, we know they 
need to be made, and we have been knowing this for a few years. 
But the problem is, an investor is not going to sink money if he 
or she doesn’t know how the money is going to get back with a re- 
turn. 

This spring, after a year and a half of hearings and workshops 
and road trips and written comments, we put out a vision for a 
wholesale power market platform. It is also known by its earlier 
name, standard market design. We recognize the regional nature 
of the power grid and the uncertainty that is created by having bal- 
kanized systems across the country. The platform would mandate 
participation in RTOs and require a transition process. 
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Many other features are left to regions to design and for State 
officials to participate in, crafting the details. But as a practical 
matter, we do need platforms upon which regional grid manage- 
ment can happen so that the benefits that we all know can happen, 
that we have seen in parts of the country where this planning has 
gone forth, can move forward. Thank you. 

Senator Voinovich. Thank you. We will limit our questions to 
each of the Members of the Subcommittee to 5 minutes and rotate 
it. 

The two of you have had a chance to observe what has happened 
here in the last several years and most recently on August 14. The 
real question that I have is, what is the best way of ensuring that 
it doesn’t happen again and that we have the reliability that is out 
there, and who should be in charge of the whole operation? 

In other words, we have had a kind of a mixed system out there. 
FERC has a role and the independent systems have a role, and you 
have the regional groups. How do we improve that system so that 
we don’t have what happened on August 14? 

Mr. McSlarrow. Mr. Chairman, I think that there are several 
different answers, but the most obvious one is to pass the energy 
bill that includes the provisions that I outlined in my testimony. 

Senator Voinovich. You believe that the provisions that are in 
the energy bill now, which is in the Conference Committee, will get 
the job done? 

Mr. McSlarrow. They will get most of the job done. What I don’t 
know and we won’t know until we conclude our task force inves- 
tigation is whether or not there is something specific to August 14 
in addition to the recommendations that I have outlined that are 
included in the Conference. 

But the energy bill in Conference has a robust electricity title. 
It deals with siting. It deals with incentives for transmission. It 
deals for encouraging the development of regional transmission or- 
ganizations. It deals with FERC’s ability to clarify those roles and 
provide incentives. It does a lot, and most importantly, it includes 
mandatory enforceable standards that are implemented by the Na- 
tional Electric Reliability Council and enforced by FERC. So I can’t 
say right now it will do 100 percent, but it will do 

Senator Voinovich. Implemented by the Reliability Council, but 
they are enforced — FERC’s role will be to enforce and make sure 
that they are carried out? 

Mr. McSlarrow. Yes, sir. 

Senator Voinovich. Which is not the case today? 

Mr. McSlarrow. Yes, sir, that is right. Today, it is a voluntary 
reliability regime and FERC has no authority to enforce that, nor 
does NERC, for that matter, and there is certainly no penalty for 
not playing by the rules. 

Senator Voinovich. Well, we are hoping you get an energy bill 
out before the end of the year. The joint U.S. -Canadian study, you 
said it is not going to be months, it is going to be weeks. How long 
do you think that it will take before we wrap that up to determine 
whether or not they come up with something that ought to be 
added to the provisions of the energy bill? 

Mr. McSlarrow. I would say we should do the energy bill as 
quickly as we can. We will get our task force completed as quickly 
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as we can. If there are recommendations and it is still timely to 
add to the energy bill, we will, but the administration would urge 
that we not slow down the energy bill. It is important to get it 
done. If there are additional recommendations, we will come and 
make those to Congress. There are plenty of vehicles that we can 
add that I am sure will have widespread bipartisan support to en- 
sure that the outage of August 14 doesn’t occur again. But we 
would urge to move forward as quickly as possible on the energy 
bill. 

Senator Voinovich. Mr. Wood. 

Mr. Wood. On the more granular level, that being the legal 
framework that would be clarified 

Senator Voinovich. Mr. Wood, I just want to say something. You 
are the Chairman of the FERC. 

Mr. Wood. Yes, sir. 

Senator Voinovich. You have, what, three members now? 

Mr. Wood. We are down to three, yes, sir. 

Senator Voinovich. OK. I understand that you are not even ca- 
pable of doing things because you need two more members? 

Mr. Wood. Oh, no. We are doing fine. 

Senator Voinovich. Are you? 

Mr. Wood. We are down two, and 

Senator Voinovich. You don’t need another two members? 

Mr. Wood. We are look forward to them being here, yes, we do. 
[Laughter.] 

We definitely do. 

Senator Voinovich. Well, I think that we ought to move as 
quickly as we can so that you have those two new members. 

Mr. Wood. One of our three, however, his term would be up 
when Congress adjourns, so at that point, we do have trouble. 

Just to follow up on Mr. McSlarrow’s comment, getting the legal 
clarifications of that world out there are certainly helpful. The un- 
certainty that is created by not knowing what the future looks like 
makes investment an almost non-event. So I do want to strongly 
urge from the Commission’s perspective, that is kind of down a lit- 
tle bit more in the trench, that we really do need the Congress to 
say this is what the energy world is going to look like. Putting it 
off for yet another session just means 2 more years of investor un- 
certainty, and quite frankly, we have had that for quite a while 
and we need to move on. 

So I do want to recommend, as Mr. McSlarrow did, that we do 
get the energy legislation out. I think it has got potential to really 
clarify a lot of issues here. Our job is to take it to the doability 
point and to take the mandatory reliability rules that NERC and 
its experts would work up and go through a public vetting process. 
If, for example, we find that those conflict with some other Power 
Act obligation, we have an obligation under the proposed bill on 
both the House and the Senate side to kick it back and say, rethink 
this because of these issues. 

We have to delicately balance this oversight role because it is 
also going to be overseen by independent Canadian authorities, and 
so this, as we saw, it is an international power grid. We have to 
make sure that — and I think the law has been written in a way 
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that is very respectful of that. So those are the kind of delicate 
things on reliability. 

The question is, OK, who is going to actually implement these 
standards on a day-to-day basis? Are we going to have it be the 130 
independent utility control areas, some big, some small, across the 
country, or are we going to try to look at that on a more regional 
basis, as we were talking about earlier in our opening statements. 
I do think that the regional model is clearly the correct way to go 
and also the common sense way to go. 

Senator Voinovich. Thank you very much. My time is up. Sen- 
ator Levin. 

Senator Levin. Thank you, Mr. Chairman. 

I would like to ask about the current authority that FERC has 
in this area of regulation. FERC indicates it is not a clear authority 
to assure reliability, but the regulations which have been issued, 
the orders which have been issued by FERC, suggest otherwise. 

In February of 2000, FERC issued the final rule, an order enti- 
tled, “Regional Transmission Organizations,” the RTO, which “codi- 
fies minimum characteristics and functions that a transmission en- 
tity must satisfy in order to be considered an RTO.” You grant RTO 
status to regional entities, such as MISO, that operate trans- 
mission lines in Michigan and Ohio. 

In the order, it says that it establishes required characteristics 
and functions for these RTOs for the purpose of promoting effi- 
ciency and reliability. Then the order goes on to state that an RTO 
“must ensure the integration of reliability practices within an 
interconnection and market interface practices among regions.” 
And it goes on to say that the RTO “must have exclusive authority 
for maintaining the short-term reliability of the grid that it oper- 
ates.” So it sounds like you have authority now and, indeed, will 
not certify or will not license an RTO unless it does those things. 

Mr. Wood. Correct. 

Senator Levin. What is lacking? 

Mr. Wood. Well, that whole program has requirements and obli- 
gations, but it is not — it is a program that utilities volunteer to be 
in. So the predicate for all those required characteristics is really 
set back in that rule, which our proposal of recent months would 
change and make mandatory for everybody. But that rule in 1999, 
which was finished up in 2000, is at its heart an encouragement 
or voluntary. So the legal authority on that was, I don’t think, a 
problem, but it was one that the Commission chose not to exercise 
at the time. 

Senator Levin. Can you refuse, then, to certify an RTO? 

Mr. Wood. We could, yes, sir. 

Senator Levin. And then what would be the penalty? Why would 
an RTO care if it were not certified? In other words, if you can in- 
sist that there be reliability standards that an RTO will adopt 

Mr. Wood. Under current law 

Senator Levin [continuing]. As part of the certification process, 
why isn’t that enough power right now to achieve those reliability 
standards? 

Mr. Wood. Where you have utilities agreeing to get into an RTO, 
you are correct. There is no question there. Where you have got 
some that do and some that don’t, or some that will and some that 
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won’t, then you do have some concerns, and that is actually one of 
the issues we have in the Midwest, is you have some utilities that 
have not joined up with an RTO and there are basically holes in 
the Swiss cheese. 

Senator Levin. The standards which are in the House and the 
Senate bill, I take it, are adequate as far as you are concerned in 
the energy bill to achieve the authority that you need to make 
mandatory these standards, reliability standards? 

Mr. Wood. Yes, sir. 

Senator Levin. OK. You, Mr. McSlarrow, the Energy Department 
supports those provisions, I understand. 

Mr. McSlarrow. We do, yes, sir. 

Senator Levin. Now why, if the energy bill — if, and I know it is 
a big if — it runs contrary to your hopes, perhaps your expectations, 
but at least your hopes — if the energy bill doesn’t look like it is 
going to pass this year, why is it not essential that we pass these 
provisions to assure reliability and that there is not a repeat of the 
blackout? Why do we hold them hostage to the rest of the energy 
bill, which has all kinds of complicated provisions where people are 
in dispute over them? We have everything from the Alaska wilder- 
ness to things which are almost as complicated as that. 

Why should we say that we are not going to do what we need 
to do to prevent a future blackout and give FERC what it needs 
in terms of authority until and unless we get all these other energy 
issues resolved? 

Mr. McSlarrow. I don’t think that it is just about NERC’s reli- 
ability standards. 

Senator Levin. About 

Mr. McSlarrow. About these mandatory reliability standards. 
And quite frankly, I have not met a utility official or an engineer 
who hasn’t treated them as if they were mandatory. So, I mean, 
making them voluntary to mandatory does, yes, put the stick a lot 
bigger, because now you can do monetary penalties. 

But at its core, there is more in the energy bill than just this and 
it is that certainty that I was mentioning in my answer to Chair- 
man Voinovich, is that we can make all the reliability standards 
in the world, but if you don’t have anybody that wants to come and 
invest in upgrading the grid, not just big new power lines. We 
might need some of those, sure. 

But investing the kind of smart technologies that are out there 
that are dying to be implemented on the grid that utilities have 
zero incentive to employ right now if they don’t understand what 
the regulatory and investment framework is going to be. And I will 
say, there are a number of provisions throughout both the House 
and Senate bills that do provide, I think, good contours for that in- 
vestment certainty for the next 10 years. 

Senator Levin. It would be less likely that they will invest? 

Mr. McSlarrow. Yes, sir, if we just have 

Senator Levin. That is fine. 

Mr. McSlarrow. If we have no bill or just a stand-alone reli- 
ability bill, I think. 

Senator Levin. I understand there would be less incentives, but 
you still need these reliability standards. There may or may not be 
those kind of investments. Those are presumably positive features. 
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But in the meantime — and you are not guaranteeing that invest- 
ment with the energy bill, with those other provisions. You are just 
presumably facilitating. 

But on this, we know there is a consensus that we need to give 
for this power to force these regional entities, these regional coun- 
cils to take the steps to make these mandatory, and if they don’t 
or if they are not adequate, for FERC to substitute its own stand- 
ards. So they will do some good by themselves. They will clarify 
your power by themselves. And so it seems to me that we are risk- 
ing needlessly, we are risking that additional clout by linking it to 
all of these other provisions. 

I am talking about all the other provisions of the energy bill. I 
am not talking about the ones that you have just outlined. I am 
talking about drilling in Alaska and everything else that is in here. 

Mr. McSlarrow. Senator, if we are at the end of session and 
nothing has happened on the energy bill, I am certainly not pre- 
pared to say today that we are going to take off the table any op- 
tions between the administration and the House and Senate lead- 
ership. 

I will say this. We have lurched from energy crisis to energy cri- 
sis, from California at the beginning of our administration to gaso- 
line prices to oil prices to natural gas shortages, back to electricity, 
to home heating oil, and every time, there is something in the en- 
ergy bill that could be taken out and passed right then. And our 
view is, we need to stop this and pass a comprehensive energy bill. 
So before we get to that point, I think we should do everything we 
can to get it done. 

Senator Levin. And if you get to that point, you are open mind- 
ed? 

Mr. McSlarrow. I am open minded, yes, sir. 

Senator Levin. Thank you. My time is up. 

Senator Voinovich. Senator Lautenberg. 

Senator Lautenberg. Thank you, Mr. Chairman. I am curious as 
to whether or not there are mandates or requirements for RTO for- 
mulation or any indication of a preference by FERC that suggests 
that this is a good way to operate. I mean, there was a mention 
of it in the statement that Mr. Wood made, that PJM functioned 
very well because we had the capacity to control the switches and 
the supply, the connections that were necessary to keep our lights 
on with a relatively minimal effect. 

So is there anything in the regulation that says or in the rules 
that say if there must be an RTO, that there should be an RTO. 
Is that what you’re thinking? Has any direction been given? Are we 
in our area just lucky that, by chance, we had formed this RTO? 

Mr. Wood. I would think one should never discount luck. I think 
the ability of the utilities actually started back in the 1920’s, Sen- 
ator Lautenberg, in the Pennsylvania, New Jersey, Maryland, and 
Delaware areas, including the District. However, yes, in 1999, the 
question that Senator Levin was asking me about, the Commission 
that I chair now did put forth a detailed set of standards that 
should be met on a voluntary basis by the utilities under FERC ju- 
risdiction throughout the country. I think the Commission at the 
time hoped that that would happen over an 18-month period. It did 
with a few utilities, but — and particularly throughout the West, 
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with the exception of California, and in some Southeastern 
States — we have not seen a lot of forward progress on coming to- 
gether on that. 

And then throughout the country, random utilities are not par- 
ticipating, either because they are not FERC jurisdictional or they 
just choose not to because it is not a requirement. So, actually, we 
have proposed, and have out there, a rule that would make that 
mandatory and that is what the subject of some debate is here in 
the Senate. 

Senator Levin. Mr. McSlarrow, last week’s House hearing on the 
blackout, Secretary Abraham indicated that the initial investiga- 
tion findings of the blackout task force will not be open to the pub- 
lic. But DOE has also publicly stated that it wants a transparent 
process. Now, what is going to happen here? Is it going to be trans- 
parent, open to the public, or is it going to be filtered, censored, or 
husbanded by the administration? 

Mr. McSlarrow. There are two different things going on. The 
first is the investigation. This is not like we are having a series of 
public policy meetings. These are actual investigators looking at 
data on computers and interviewing operators of the grid, operators 
of the ISOs. This is the same thing that was taking place all over 
America with any law enforcement agency. Normally, you don’t 
have the public traipse along with you doing that. 

Once they get the findings, then we actually move to a policy 
phase where we are thinking about recommendations. Secretary 
Abraham did testify, and I know it is his intention, that we will 
have as open and transparent a process as possible. 

Senator Lautenberg. I hope so, because the public is a partner 
here. We hear grumblings. People say, well, it is going to cost more 
for the users, for the ratepayers to get this thing into shape. But 
then, in my view, we have to take a look back, see what the oper- 
ating results were of these companies, what did they do with their 
reserves, did they build any reserves, and why didn’t they move 
ahead on some of these things. 

Mr. Wood, you talk about state-of-the-art digital switchings, 
things that I think are dying to be used in the current system. 
Well, if that is the case and we have technology, we understand 
what it is that would keep things going, then why haven’t we 
moved? Was there such an overwhelming profit motive for this 
commodity material that they just said, well, the heck with that. 
We are not going to do it until we are forced to do it. Where would 
the automobile business be if they didn’t improve the design on a 
constant basis? Where would other industries be? 

So what happened here? Why weren’t these investments made, 
do you think? 

Mr. Wood. From the FERC side, we do get all the financial data 
and we are looking from really tracking all the utilities involved in 
this blackout from 1990 forward to get an idea of what kind of in- 
vestments were made. I don’t have any data at this time to share, 
but we will be including that with the broader study going on here 
to just understand what sort of investment has been made in the 
transmission grid by the current people doing that and get a better 
understanding why that went up or down. But I don’t have a spe- 
cific answer to your question today. 
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Senator Lautenberg. I can tell you that the ratepayers will not 
joyfully write checks and say, OK, well, we have to fix the systems. 
You said something. Investors want to know what is going to hap- 
pen if they put money in there and they have a right to know that. 
There are opportunities for financing, especially in periods like this 
when the cost for money is relatively low, and I would hope that 
the target isn’t the ratepayer who is going to be asked to pick up 
the load because the system was faulty. 

Do either of you think that if we had RTOs in place, more RTOs 
in place, that wouldn’t it have been possible to prevent the black- 
out from occurring? 

Mr. McSlarrow. Go ahead. 

Mr. Wood. I want to resist the urge to just say yes and be quiet 
there, but 

Senator Lautenberg. No, don’t. 

Mr. Wood [continuing]. I do think some aspects of RTOs were 
very helpful. I mean, there is an understandable debate about what 
energy markets should do and what they should look like. Should 
they be like PJM or should they be something less than that? And 
that is a fair debate to have. We have it a lot. 

But at the core, the RTO is a transmission operator of a regional 
grid that recognizes what the laws of physics have told us long ago, 
that this product is going to flow where the path of least resistance 
lies, not where State boundaries or utility boundaries lie. So at its 
core, you have got that business plan going on there. 

And so to the extent an RTO does bring together and get invest- 
ment and get the kind of real-time control systems where you can 
see, in fact, that that line is out, that line is on, this one is sagging, 
to have that information come in at one time over a broad area, 
not just for one small utility but for the entire area, yes, I think 
had we had some of the utilities in the Northern part of Ohio all 
part of the same interconnected and real-time communicated grid, 
that might have been a different outcome, sure. 

Senator Lautenberg. Yesterday, I was visited by several em- 
ployees of First Energy and they reported to me that the company 
is using power transmission lines at well over capacity and has 
failed to replace sections of an old unreliable infrastructure. Can 
FERC play a role to address this kind of irresponsible part of be- 
havior on the part of a utility? 

Mr. Wood. I think we can. As I mentioned, we are talking to 
folks of that nature. I am not sure if it is the same ones as part 
of the broad investigation with the task force. And if there are spe- 
cific issues that relate to FERC’s mandate as opposed to that of the 
Canadians or the other American agencies, we will pursue those 
independently. 

I do think this is probably an area where mandatory reliability 
standards as opposed to the voluntary ones makes a difference — 
and the main difference there is you can put a price tag on vio- 
lating a mandatory standard. If a mandatory standard is, you can’t 
exceed 105 percent of the capacity more than ten minutes out of 
a day, or whatever the standard would be, then that has a financial 
consequence attached to that. 
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We do not have that today. Now, a State may have that author- 
ity. Some States do and some don’t. But we clearly do not. That 
would change with the reliability legislation in both chambers. 

Senator Lautenberg. Mr. Chairman, I will conclude with that, 
but I would hope that we go further in developing information be- 
cause it is very hard in this kind of a fairly short burst to get the 
data that are needed. I think that there is an opportunity — should 
be an opportunity for us to thoroughly review the past performance 
of these firms, to have it done and presented to us or to the Con- 
gress generally, about what took place when things were better and 
whether there was any preparation at all for the expansion that 
was inevitable as our population grew and demands were increas- 
ing. 

So I commend you for doing this, but I think that there is a lot 
more that we have to learn before we are satisfied that it is not 
going to be the ratepayers doing this. 

Senator Voinovich. One of the things that I have mentioned is 
that it is my intention that once the report is finished, to have 
hearings on that report. It is going to be public. We are going to 
have people in here and we are going to go over the report. I would 
rather wait until everybody has got all the information to do it 
rather than do it prematurely, as is so often the case around here. 
Sometimes, we never do get to the real cause of something because 
you have tackled it before people have had the facts before them. 

I would like to thank the panel very much for being here today. 
I just want to make one thing clear again. Both of you feel that 
to remedy some of the things that we have talked about here, par- 
ticularly Senator Lautenberg about what people should do and 
shouldn’t be doing. You believe that the language in the energy bill 
electricity title gets the job done. 

Mr. McSlarrow. We do. 

Senator Voinovich. That is what I am worried about. I want to 
make sure that since this new incident on August 14 — that there 
isn’t anything new that has come to the surface that ought to be 
reflected in that language so it is as comprehensive as possible. 

I would also like to mention that I happen to believe that this 
should stay in the energy bill. It is long overdue that this country 
have an energy policy. There is so much uncertainty out there, not 
only in this area, but also what utilities can do to reduce their NO x , 
SO x , and mercury emissions, and it is a chaotic situation of law- 
suits. It is a maze, as a matter of fact. We need to clear that out 
and let everybody know where they stand, and if they get out of 
line, that they are going to pay a steep penalty for getting out of 
line. 

I hope that you keep staying with this and keep the administra- 
tion on it, that we have to get an energy bill out and we have to 
get it out as soon as possible. Thank you very much for being here 
today. 

Mr. Wood. Thank you. 

Mr. McSlarrow. Thank you, Mr. Chairman. 

Senator Voinovich. We are going to recess for a few minutes. 
There is 4 minutes left in the vote. So we are going to go over and 
vote and then we will stick around for the vote on the next bill and 
then come back and we should have about 35 or 40 minutes so that 
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maybe we can get the testimony from our witnesses that are here 
today. Thank you. 

[Recess.] 

Senator Voinovich. The Subcommittee will come back into order. 
I want to apologize to the witnesses. We never know in the Senate 
what is going to happen and we had some stacked votes then. We 
are going to try and get as many of you in as we possibly can. I 
have to go back. I have about 20 minutes between now and then 
and I will try to have the next vote, which is very important be- 
cause it is an amendment that I am cosponsoring, and then we will 
see how it works out. 

You all know who you are and I am glad that you are here. For 
the record, on our second panel we have Dr. Alan Schriber, who is 
Chairman of the Public Utility Commission of Ohio. Thank you for 
being here, Dr. Schriber. 

Next to Dr. Schriber, he has his predecessor, Craig Glazer, who 
was former Chairman of the PUCO of Ohio, and now is Vice Presi- 
dent of PJM, and someone that has worked with me on energy 
issues since he was in the Water Department of the City of Cleve- 
land and we rewrote the public utilities law of Ohio. And then 
when I became governor, I made him Chairman of the Public Utili- 
ties Commission, and Craig, I am glad that you are here. 

James Torgerson is the President and CEO of the Midwest Inde- 
pendent System Operator. William Museler is the President and 
CEO of the New York Independent System Operator. 

And rounding out the panel, James Kerr, a Commissioner of the 
North Carolina Utilities Commission, and Dr. Mark Cooper, the Di- 
rector of Research at the Consumer Federation of America. 

I wish the witnesses would stand up and I would swear you in, 
as is the custom. 

Do you swear your testimony is the whole truth and nothing but 
the truth, so help you, God? 

Mr. Schriber. I do. 

Mr. Glazer. I do. 

Mr. Torgerson. I do. 

Mr. Museler. I do. 

Mr. Kerr. I do. 

Mr. Cooper. I do. 

Senator Voinovich. The record will show that they all answered 
in the affirmative. We will start out with Mr. Schriber. 

TESTIMONY OF ALAN R. SCHRIBER , 1 CHAIRMAN, PUBLIC 
UTILITY COMMISSION OF OHIO 

Mr. Schriber. Thank you, Mr. Chairman, and thank you for the 
opportunity to be here. My testimony, I won’t read. If it is sub- 
mitted for the record, that would be great. 

Senator Voinovich. May I say something? You all heard the tes- 
timony before from the other two witnesses, I think, didn’t you? 

Mr. Schriber. Yes. 

Senator Voinovich. And you also heard the questions from Sen- 
ator Levin and you also heard the questions from Senator Lauten- 
berg 


lr The prepared statement of Mr. Schriber appears in the Appendix on page 88. 



21 


Mr. Schriber. Yes, sir. 

Senator Voinovich [continuing]. And you heard some of my 
questions. If you want to sprinkle in some of your reactions to that, 
I would be very grateful. 

Mr. Schriber. OK, sir. 

Senator Voinovich. I am particularly interested in whether or 
not the language in the energy bill, which I am sure most of you 
are familiar with, is adequate to get the job done or if you have 
some problems with that language. I am going to be very much in- 
volved, and have been, with that bill, and I would sure like to hear 
from any of you if you think they have left out — there is a big hole, 
or there is something in it that we feel goes too far or whatever 
your opinion is. 

I want to fix the problem, and we will talk about this investiga- 
tion after they get the job done. But we have this wonderful oppor- 
tunity to make a difference and I want to make sure that we don’t 
miss this opportunity. 

Mr. Schriber. 

Mr. Schriber. Thank you very much, and I will note that I am 
a member of that binational task force and I would look forward 
to delving into that and hopefully someday reporting back to you. 

Just to get to the point, which I know you want to do, we want 
to talk about reliability for a moment because I think everybody 
agrees that reliability is a critical issue. I think for clarity, we need 
to understand that reliability can take on different meanings. In 
the arcane world of electricity that we deal with, we talk about reli- 
ability in terms of security and resource adequacy. I think what we 
are talking about now, in light of the blackout of August 14, is the 
physical properties of the network, of the system, of the grid. How 
secure is it? Is it reliable? Is it going to break? I think that is really 
important. 

I don’t think we have a third world system. I think what we do 
need is rules and we need standards. We need NERC and FERC 
to have the authority to promulgate and enforce those rules, and 
I think the States can play a very prominent role in enforcement. 

As you know, having been governor, Mr. Chairman, in Ohio, as 
in other States, we enforce — the State enforces rules that are pro- 
mulgated by other Federal agencies, highway rules, rail rules, and 
so forth, and there is a good argument that can be made for having 
the ability to enforce rules that are for FERC, if you will. But I 
think, again, we all agree that is very necessary and I would say 
that the reliability provisions of that electricity title are absolutely 
essential. 

I also agree with Chairman Wood that optimum allocation of re- 
sources, of dollars toward the system, to the extent that it needs 
to be fixed, and again, I don’t think it is a third world system. I 
think it is like a highway system. It is broken down in some places. 
It needs to be fixed in others, and in some cases, congestion needs 
to be taken care of. I think dollars need to flow where they need 
best be invested and I think this is done only if you have a control 
which takes place over a larger area than among 12 fragmented 
transmission systems, as we have in the Eastern Interconnect. 

So, therefore, I would urge you and your colleagues to give FERC 
the authority they need to move forward in establishing these 
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large, centralized transmission systems that embrace not just one 
or two, but a very large section of the Eastern Interconnect. I think 
that gives FERC the ability to put these organizations in place, and 
within those organizations, I believe decisions can be better forth- 
coming. 

Senator Voinovich. Can FERC right now order a utility into an 
RTO? 

Mr. Schriber. No, but the States in some cases have, where 
mergers have taken place. As merger agreements or as provisions, 
like in Ohio’s law, States — rather, utilities were ordered to join 
RTOs. 

Senator Voinovich. So that right now, the power is in the States 
to get utilities to join RTOs, and this legislation would give FERC 
the power to order them in? 

Mr. Schriber. That is correct. 

Senator Voinovich. OK. 

Mr. Schriber. We would hope. The States said, at least our 
State has ordered that, and it has been a provision some mergers. 
At any rate, I think FERC should be supported in their endeavors. 

And, I think those are the two main provisions. We could talk 
about the Public Utilities Holding Company Act. I would have no 
problems with that being suspended or rescinded. I think that we 
are likely to hear some arguments in opposition. They have to do 
with deregulation as being a problem. And I am prepared, although 
I won’t go into it now, but upon questioning, I would be prepared 
to take issue with that. I don’t think deregulation has much, if any- 
thing, to do with what is going on. 

I think with that, I know you want to move on and I know you 
want to hear what others have to say with respect to the questions 
that were raised here, so I will conclude with that. 

Senator Voinovich. Thank you. Those comments were wonder- 
ful. 

Mr. Schriber. Thank you. 

Senator Voinovich. Mr. Glazer. 

TESTIMONY OF CRAIG A. GLAZER , 1 VICE PRESIDENT, PJM 
INTERCONNECTION, L.L.C. 

Mr. Glazer. Thank you, Mr. Chairman. It is always tough to, 
first off, follow your successor. As bad as August 14 was for me 
that night, I actually felt good that after 10 years, I said, somebody 
else is in charge of the PUCO, not me for a change. So I am glad 
I didn’t have his job that night. 

I also, Mr. Chairman, I have to tell you, I sleep better at night 
knowing that we have a member of the U.S. Senate that actually 
ran an electric system, and not only ran an electric system, but ran 
it and made it competitive and made the model of competition 
work. I mean, competition worked in the City of Cleveland. It 
brought savings to the residential customers and it was used by 
you as an economic development tool. 

I think those lessons are really important, so I just wanted to say 
thank you for your leadership over the years. It is important that 
we have somebody in the Senate that actually had hands-on experi- 


1 The prepared statement of Mr. Glazer appears in the Appendix on page 98. 
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ence, and I think that means a lot as we move forward with this 
and I appreciate your involvement in this energy bill because it is 
so important to a State like Ohio. 

Let me cut quickly to the chase. I remember one day back in 
1996 in the dusty halls of the Ohio legislature talking with the 
then-chairman of the State Committee, Senator Richard Finan, 
about this electric restructuring. I remember discussing with him 
electric restructuring and saying, “If this is about instant gratifi- 
cation, forget it. It is not going to happen. If it is about cutting 
rates, you don’t need to restructure the industry. I have got enough 
authority at the PUCO to do that right now. If it is about command 
and control from the government and telling people what to build, 
we have the authority to do that.” 

But if we needed to attract investment, when investment is crit- 
ical to this industry, we need, and I firmly believe we still need, 
to reform and continue to reform the structure of this industry. The 
electric industry is the most capital intensive industry other than 
the military, other than the military, the most capital intensive in- 
dustry in the world. And right now, we are teetering in a very dan- 
gerous place. 

And you asked about the provisions of the energy bill. There are 
good provisions, but there is some language floating around that 
we think would actually might set this industry back if we are not 
careful. I think we are in kind of a difficult place and the investors 
are watching the fact that we are in a difficult place. Let me come 
to that just in a minute. 

I know Dr. Cooper, who I know very well and respect, is on this 
panel and he is going to say restructuring doesn’t work. We should 
just go back to the old way. Well, as I said, Mr. Chairman, at the 
beginning, you ran a competitive electric system. You did make it 
work. By the same token, in PJM, we have actually been able to 
make the system work on the wholesale level. We have seen great- 
er efficiency in generation. People are better maintaining their 
equipment than they did under the old regulated system. We have 
been able to attract new investment. We have been able to keep 
prices stable. 

And what is important for Ohio, in particular, but also for Michi- 
gan in this, we have been working with our counterparts in the 
Midwest ISO on a joint operating agreement and reliability plan. 
I am not here to say that would have been the total panacea, but 
it addressed a number of things that went afoul the day of August 
14. Had that been in effect, I think we would have certainly re- 
duced the number of people involved. We would have had clearer 
rules in place. 

I think it is important as First Energy moves into the Midwest 
ISO, which it has chosen to do, as AEP moves into PJM, as it has 
chosen to do, that we have that agreement in place. I hope we can 
move forward with the Federal Energy Regulatory Commission, the 
NERC, the National Electric Reliability Council. It will help Ohio 
and it will provide a new level of reliability in the Midwest. To the 
credit of Midwest ISO, we have been working together on that, 
really had signed that, had that in place, at least conceptually, and 
we are going through the stakeholder review process. 
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You had correctly asked, what can Congress do? The first thing 
I would say is do no harm. There are provisions floating around, 
because this country is very split on the very issues you are talking 
about, do we mandate RTOs? Commissioner Kerr will say, my re- 
gion doesn’t need RTOs, and I respect him for saying that. But we 
run into the law of unintended consequences. There is language 
floating around in the Senate Committee draft that would really tie 
FERC at the knees, would ban its efforts on — delay its efforts on 
standard market design. 

Mr. Chairman, I would rather almost kill something than delay 
something. Delay, frankly, is an easy cop-out, but delay is really 
the death knell to investment and I am very worried about that. 

Also, we heard a lot about investing. You mentioned in your 
opening statement, invest in transmission. We should do that, but 
let us do it wisely. 

One of the things which we do at PJM, one of the things which, 
frankly, we did at the PUCO and Dr. Schriber continues to do is 
regional planning. We ought to not just throw money at trans- 
mission. We ought to integrate it with regional planning. Back at 
the time, we had a big issue, you may recall, about scrubbers. 

Senator Voinovich. Mr. Glazer, do you want to wrap it up? 

Mr. Glazer. Let me wrap it up by saying, at the end of the day, 
we need to pause and study what has happened, but we do need 
to move forward. We need to respect and go at different paces for 
different regions. But let us not cut the Federal Energy Regulatory 
Commission at the knees. Let us give it the continued ability to get 
the job done, and that is in play right now in the energy bill. 

Senator Voinovich. Both you and Mr. Schriber feel that the leg- 
islation, the electricity title, gets the job done, but you are worried 
about some stuff floating around that would delay it? 

Mr. Glazer. Yes. 

Senator Voinovich. That is what you are — the language is OK, 
but you are worried about what is going on inside the Committee? 

Mr. Glazer. Yes, exactly. 

Senator Voinovich. OK. I have got it. Mr. Torgerson. 

TESTIMONY OF JAMES P. TORGERSON , 1 PRESIDENT AND 

CHIEF EXECUTIVE OFFICER, MIDWEST INDEPENDENT 

TRANSMISSION SYSTEM OPERATOR, INC 

Mr. Torgerson. Good morning, Mr. Chairman. Thank you for in- 
viting me to this hearing. I am Jim Torgerson, President and CEO 
of the Midwest ISO. I want to provide some insights today on what 
I saw concerning the circumstance surrounding the power outages 
of August 14 and offer suggestions as to what might be done in the 
future. 

The Midwest ISO was formed in 1998. It is the Nation’s first vol- 
untary regional transmission organization that did not originate 
from a legislative mandate or against the backdrop of a tight power 
pool. The Midwest ISO is also the first entity found by the Federal 
Energy Regulatory Commission to be a regional transmission orga- 
nization. 


lr The prepared statement of Mr. Torgerson appears in the Appendix on page 111. 
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The Midwest ISO’s region covers portions of 15 States and the 
province of Manitoba, and our current role is that of a NERC-cer- 
tified reliability coordinator. As a reliability coordinator, the Mid- 
west ISO monitors flows on key transmission facilities, develops 
day-ahead plans, conducts next-hour analyses regarding the high- 
voltage grid, and communicates with the control areas in our re- 
gions who have the primary control capabilities to open and close 
transmission circuits and to redispatch generation. 

Three of the more than 30 companies within our reliability terri- 
tory suffered outages in the blackout, Consumers Power Company, 
Detroit Edison, and First Energy. The cause of the blackout and 
why it cascaded will be forthcoming from the work being done by 
the international task force formed by President Bush and Prime 
Minister Chretien of Canada. The Midwest ISO only has a part of 
the data needed to reconstruct those events, and in addition to ap- 
pearing at today’s hearing, the Midwest ISO is cooperating with 
the international task force and the General Accounting Office in 
determining what occurred on August 14. 

Likewise, the reason for the cascading effect of the outage is un- 
known at this time. The analysis that has been done to date in the 
Midwest seems to indicate there were a number of events in the 
Eastern Interconnection on August 14. Some are surely related to 
separations and the substantial losses of load that occurred and 
others are likely unrelated. 

At approximately 4:10 Eastern Daylight Time, portions of the 
Eastern Interconnection were separating from one another 

Senator Voinovich. Mr. Torgerson, I know that and I appreciate 
it. 

Mr. Torgerson. OK. 

Senator Voinovich. I would really like you to — you have the 
Midwest. You have seen the language in that energy electric title. 
Do you like it or don’t like it? Do you think they are stepping on 
your toes? Do you think that utilities should be able to be man- 
dated join? That is what I am interested in. 

Mr. Torgerson. Yes, Mr. Chairman. We find that we need 
strong reliability standards, and they should be mandatory. We are 
comfortable with the language in the energy bill. I think it does 
provide what we need for us to go forward. I think the energy bill, 
overall, will be satisfactory. I agree with my colleagues here about 
the energy bill as it stands. 

The other thing I wanted to point out is the Midwest ISO has 
formed an organization, the Organization of MISO States. The Or- 
ganization is composed of all the State commissions within the 
Midwest ISO that have gotten together to work cooperatively with 
the Federal Government on siting of transmission facilities. We be- 
lieve that approach holds great promise in allowing the siting of 
needed transmission facilities and protecting the regional efforts to 
address issues associated with the wholesale electric market. 

Senator Voinovich. So in terms of the siting of transmission 
lines, you would be involved in doing that, in other words, setting 
priorities as to where the transmission lines might be, looking at 
your grid and how it all works out? 

Mr. Torgerson. In our overall approach, we provide a long-term 
transmission expansion plan that covers the entire footprint. We 
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just had it approved by our board. We work with all the State com- 
missions in getting this built up. Then we do a review of it to make 
certain that what is in the plan will relieve congestion. It is not 
overstepping things, it is making sure there are no duplicative in- 
vestments, and it looks at how we are solving problems in the Mid- 
west. That is then worked with all the States and the State com- 
missions and then ultimately approved by our board, and our board 
approved $1.8 billion in investments that needed to be made for re- 
liability in the Midwest over the next 5 years. So those are the 
things we have been focused on. 

Senator Voinovich. Do you think that the provisions of the elec- 
tricity title in terms of siting, dealing with the NIMBY problem 
and the NEPA problem, are taken care of, or do you still think 
more is needed. I mean, one of the problems is, how do you pay 
for it, and two, how can you get it done, and do you think that the 
language is adequate enough so that if you decide these trans- 
mission links have to be sited, that it can be done? 

Mr. Torgerson. We think it is adequate. I think the approach 
we are taking with the Organization of Midwest ISO States is they 
are looking at it regionally, but they are also breaking it down. If 
a project is going to affect two or three States, those two or three 
States will then work with us on getting the siting done, and that 
is the game plan behind it. So we think it can be effective. 

Senator Voinovich. Thank you. I want to make clear also to you, 
and I again apologize to you, because of this vote schedule, your 
full testimony will be entered into the record. 

Mr. Torgerson. Thank you, sir. 

Senator Voinovich. Mr. Museler. 

TESTIMONY OF WILLIAM J. MUSELER , 1 PRESIDENT AND 

CHIEF EXECUTIVE OFFICER, NEW YORK INDEPENDENT SYS- 
TEM OPERATOR 

Mr. Museler. Thank you, Mr. Chairman. I will skip most of my 
prepared remarks and go to the subjects that you wanted us to con- 
centrate on here. 

Just a brief background. The New York ISO began operation in 
1999. We are responsible for operating the grid, assuring open ac- 
cess, and operating New York’s electricity markets. 

With respect to the policy 

Senator Voinovich. May I ask you something? Are you part of 
PJM? 

Mr. Museler. No, sir. 

Senator Voinovich. You are a separate operation? OK. I have 
got it. 

Mr. Museler. That is correct. The Northeast consists of — and we 
all operate approximately the same way. PJM, which is the largest 
RTO. There are three ISOs, New York, New England, and Ontario. 
So when we talk about the Northeast, we typically talk about those 
four ISOs and RTOs which all operate more or less the same way 
and the markets are — at least the New York, PJM and New Eng- 
land markets are very similar to one another. 


lr The prepared statement of Mr. Museler appears in the Appendix on page 118. 
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With respect to the policy recommendations, it is certainly strik- 
ingly clear that we need mandatory reliability rules. Our view is 
that NERC should be the standard setting authority under FERC 
jurisdiction. 

You asked, however, whether or not there were — whether or not 
provisions in the pending legislation are adequate. As you know 
better than I do, there are differences between the House and Sen- 
ate versions. For example, with respect to reliability, one of them 
allows for a region or a sub-region to have more stringent reli- 
ability rules if they so desire. We think that is critical. New York 
City currently has more stringent reliability rules than the NERC 
rules and we think that if the areas, the States, and the operating 
authorities believe that more stringent rules are required in certain 
areas, that we should be permitted to do that. 

It would not, in my view, be acceptable to have an area like New 
York City be held to the floor of reliability when the importance of 
maintaining power in New York City has effects nationally and 
even internationally. So we certainly think the legislation deals 
with mandatory reliability rules very well, but there is a difference 
between the bills and we think that needs to be taken into consid- 
eration. 

With respect to siting, again, I think the legislation is very good 
in that regard. But with respect to backstop authority for siting of 
transmission lines, there is a difference. So it depends on what the 
Conference Committee comes up with there. The States have the 
primary responsibility for siting transmission lines. I don’t think 
anybody disagrees with that and State compacts, State agree- 
ments — and PJM is a good example of that in terms of the agree- 
ments they have with their States to move on their transmission 
plans and actually build things — is a good example of that. 

However, should the States fail, there, in our judgment, needs to 
be some backstop to ensure that the public interest is taken into 
consideration. 

Senator Voinovich. Mr. Museler, I am going to have to excuse 
myself because I have got 2 minutes left on the vote, but the point 
you are making is that if the States don’t do the siting and the 
siting is needed then the backstop should be, what, FERC? 

Mr. Museler. Yes, sir. 

Senator Voinovich. FERC should be able to say, these lines have 
to be sited. We are looking at the big picture. It has to be done. 

Mr. Museler. Well, they should have the authority to make that 
judgment. 

Senator Voinovich. To make the judgment. 

Mr. Museler. They may affirm the States. 

Senator Voinovich. OK. I think I will try and be back in about 
10 minutes and we will finish up, and then maybe give you all an 
opportunity to share some more with me. Thank you. 

[Recess.] 

Senator Voinovich. The Committee will come to order. 

Mr. Museler, you had some time left. Do you want to make any 
last one or two comments? 

Mr. Museler. I would just make one additional point, Mr. Chair- 
man, and that is with respect to the cost recovery, transmission 
cost recovery provisions in the legislation. It does provide FERC 
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authority for certain cost recovery measures. I would note that I 
would not suggest anything additional except that even with that 
authority in States and jurisdictions that have bundled trans- 
mission rates and rate caps, the fact that FERC can set a higher 
transmission rate does not translate into the actual entities — in all 
cases, it does not translate into the entities actually being able to 
get that as an incremental amount of revenue recovery. 

So FERC needs the authority, in my judgment, to be able to en- 
sure that its incentive rate of returns or its regular cost-base rate 
of returns actually are able to flow through to the transmission 
builder-owner within a reasonable amount of time. If there is a 5- 
or 10-year rate cap that prevents that — that will chill investment 
if investors know they can’t even begin to recover for 5 or 10 years. 

That is all I would like to add, sir. 

Senator Voinovich. Thank you. 

Senator Voinovich. Mr. Kerr. 

TESTIMONY OF JAMES Y. KERR, II , 1 COMMISSIONER, NORTH 
CAROLINA UTILITIES COMMISSION 

Mr. Kerr. Thank you, Mr. Chairman. My name, again, is Jim 
Kerr. I am with the North Carolina Utilities Commission, and to 
my knowledge, I am the first State official from outside the directly 
affected region and I appreciate your Committee, Subcommittee’s 
interest in hearing to some extent from those of us beyond the di- 
rectly affected area. 

I have filed written testimony which is part of the record, and 
given the time constraints will also dispense with my prepared 
statement this morning. What we have tried to do in our testimony 
is to describe that we have a very different electric industry struc- 
ture in the Southeastern United States that is dependent on 
vertically integrated utilities and a cost-based State regulatory sys- 
tem. 

With respect to reliability, I have also illustrated how that elec- 
tric industry system on a regional level is coordinated through the 
Southeastern Reliability Council and sub-regions within the South- 
east and how we deal directly with accountability, planning, coordi- 
nation, and operational control. 

With respect to the broader issues that are being discussed as 
possible reactions to August 14, in my testimony, I express the sig- 
nificant concern that regulators in my region have had with man- 
datory RTOs and standard market design initiatives at the FERC 
and then try to comment briefly on the discrete issues in the bill 
which we think, or which I think, as a personal opinion, are pretty 
good ideas. 

You have asked this morning for our thoughts on the specific lan- 
guage that is before the Congress in the Conference Committee, 
and for the sake of time, I will run through very quickly that it is 
my opinion that the Federal enforcement authority over reliability 
standards is certainly an appropriate step that this Congress could 
take. I believe that appropriate backstop siting authority, similarly, 
is an appropriate step that this Congress could take. 


lr The prepared statement of Mr. Kerr appears in the Appendix on page 127. 
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I want to point out here that on that point, I differ with the posi- 
tion of the National Association of Regulatory Utility Commis- 
sioners. They are opposed, and I feel obligated to point that out. 
It is my personal opinion, however, that the language on this issue 
in the bill is, in fact, appropriate. 

I think that appropriate incentives for transmission investment 
at the Federal level, as they are contemplated in the bill, seem to 
be appropriate. I am concerned that in January of this year, the 
FERC issued an incentive rate provision that seemed to me to 
apply incentives to the moving around of existing transmission as 
opposed to applying simply to new investment. I think if you apply 
incentive rates to rearranging the control or ownership of existing 
transmission as a way to incent folks to join regional transmission 
organizations, you are creating no new transmission and, in fact, 
are creating additional costs that will ultimately be borne by the 
ratepayers. 

Finally, with respect to what role the FERC should have with re- 
spect to regional markets and RTOs, I think it is imperative that 
Congress not allow FERC to move forward with mandatory RTOs. 
I believe that the administration said this morning they were in 
favor of voluntary RTOs and we believe that should be codified in 
the energy bill. Market design concepts, market oversight, we be- 
lieve all of that should be — the various regions of the country 
should go forward in a voluntary nature so that they can craft 
those types of solutions to the industry structure that may exist, 
whether it be in the Midwest, the Northeast, or in our area. 

When you ask, am I supportive of the energy bill itself, with the 
more discrete provisions on siting, reliability standards, investment 
incentives, that language, as I understand it in the two versions of 
the bill, seems fine to me. 

With respect to standard market design, I am not quite sure 
what the Senate version is right now. I will tell you that the provi- 
sion in the Senate, the Domenici substitute, appears to me to be 
appropriate. 

I want to just take a very brief time to respond to some of the 
concerns raised by the representatives of PJM as well as my col- 
league, Dr. Schriber, as to whether or not that language — I think 
that is the language that they were saying would somehow cut 
FERC off at the knees. As I read Section 1122 of the Domenici sub- 
stitute, it speaks to no final rule of general applicability within the 
scope of the proposed SNB rulemaking could go into effect until a 
certain period of time. 

I believe this, and I believe it would be — I read that to mean that 
if in PJM or if in MISO, that organization, the stakeholders in that 
organization and the Federal regulators can reach agreement on 
market design, FERC can certainly approve such a proposal. So I 
don’t believe that language was intended or, in fact, does constrict 
the ability of my colleagues from other regions of the country to 
move forward. That is not my intention, and to the extent it might 
do that, I think that we — and we, in fact, have offered our col- 
leagues from the Northeast and the Midwest to help craft better 
language, if you will. 

So if that was the piece that was referred to as handcuffing 
FERC’s ability to move forward, as I read the actual language, and 
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this is the only language I have seen, I don’t believe it does that. 
I believe it says a rule of general applicability, which I would think 
would mean a notice of proposed rulemaking that would be applica- 
ble across the country. 

So with that, my time is up and I thank you for the opportunity. 

Senator Voinovich. Thank you, Mr. Kerr. Thank you for your 
perspective. Dr. Cooper. 

TESTIMONY OF MARK N. COOPER , 1 DIRECTOR OF RESEARCH, 
CONSUMER FEDERATION OF AMERICA 

Mr. Cooper. Thank you, Mr. Chairman. You have asked us to 
get to the point and I have got that reputation. [Laughter.] 

In my opinion, the deregulation provisions of the legislation go 
too far and the reliability provisions do not go far enough. Let me 
first lay out the conditions why and then I will go to the specifics. 

Electricity is unique. It is not just a commodity, and we must 
never forget that. It has no substitutes. It is not storable. It is es- 
sential to public health and safety, to daily activities. It is delivered 
under incredibly demanding conditions that are extremely capital 
intensive. 

And deregulation and restructuring have increased the stress on 
the grid — so you have to recognize that — by causing a dramatic in- 
crease in the number and complexity of transactions for which this 
system was never built. It creates difficulties in coordination and 
planning as competition and contracts replace centralized decision 
making. 

PJM was a tight power pool for 50 years or so before it became 
an RTO. The RTO had nothing to do with its ability to control its 
area. 

Deregulation certainly short-circuited utility incentives to invest 
in transmission because the private interests of facility owners 
come into conflict with the shared public nature of the trans- 
mission system. It is a highway, not a market, and especially when 
you are asking them to make investments that they — for a system 
they share with their competitors. It is very difficult. 

And moreover, deregulation undermines the ability to account for 
social and environmental questions and constraints. The social cost 
of transmission is much higher than its mere economic cost. The 
fundamental problem with transmission is not inadequate incen- 
tives to invest. Utilities were willing to do so before deregulation. 
The problem is public resistance to building additional trans- 
mission facilities for environmental, health, and safety reasons. 

For these social reasons, scarcity of transmission in an economic 
sense is likely to be a permanent part of this industry’s landscape. 
That is what our people tell us. 

The benefits of the shared transmission facilities are difficult to 
allocate. This is a network that is shared. The problem is geo- 
graphic and intergenerational. Today’s investments deserve a long- 
term, long-distance transaction, maybe tomorrow’s core for serving 
native load. 

Now, I understand the pressure to do something in the wake of 
the blackout, but when it comes to electricity, doing just anything 


lr The prepared statement of Mr. Cooper appears in the Appendix on page 187. 
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will not help. You have to do the right thing or you will make mat- 
ters worse. 

Right now, you do not need to repeal the Public Utility Holding 
Company Act to improve the reliability of the system. I don’t need 
utilities going into non-utility businesses and creative massive 
multi-state holding companies that escape regulation in order to 
improve reliability. We do not need to impose the standard market 
design. And the regional transmission organizations that are em- 
bodied in it are the wrong ones to create. They are dominated by 
industry, they preempt local accountability, and they have forced 
utilities into markets for allocating transmission resources with no 
assurances that the capacity is adequate today, additional capacity 
will be built or maintained. 

We must not rely on industry self-regulation. The proposal to 
move from voluntary self-regulation to mandatory self-regulation 
misses the point. The difficulty is not the voluntary versus the 
mandatory. It is the “self’ part. We need clear accountability to 
public authorities. 

Do not create private transmission monopolies. Transmission is 
a natural monopoly, part of a shared network. Transferring control 
to unregulated companies will simply allow them to increase their 
profit and exploit their market power. 

So that is what you shouldn’t do. What should you do? I person- 
ally believe we need transmission organizations, but they have to 
be organized on a very different model than has been contemplated 
and proposed. Any transmission organization must be based on 
fairness and public accountability. Fairness requires a process for 
representation of all interests affected by transmission projects. 
The way to overcome social resistance to transmission projects is 
to give people a fair chance to present their case, defend their in- 
terest. That is what federalism is all about. It is an ugly, tough 
process, but it works because it empowers the people. 

Accountability demands that the local officials who get the phone 
calls when the lights go out are the people who are making the de- 
cisions, who have the ultimate authority. They didn’t call the 
FERC when the lights went out in Ohio. They called the Ohio 
PUC. The Ohio PUC must have a fair representation in this proc- 
ess. 

Accountability also requires transparency. We cannot have this 
conflict between the FERC and the DOE and the private companies 
and the NERC over who has got the data and who is responsible 
for the analysis. 

Finally, even if economic incentives were a problem, and I don’t 
think they are, the solution is not to increase the rate of return but 
to lower the risk, and that is what the utility model used to do. It 
established a long-term commitment. It established a stable envi- 
ronment. And frankly, all of the people who say we can’t raise 
money in the industry are living in the dot-com 1990’s, not the 
post-bust market. Give me a stock that offers a stable dividend, a 
slow and long-term growth rate, the widow and orphan stocks that 
the utilities used to be. They will have no trouble raising capital. 
But it is public policy that must create that environment that will 
promote the investment. Thank you. 
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Senator Voinovich. Thank you. You have all had a chance to 
hear each other today. One of the things I like to do is to give wit- 
nesses an opportunity to comment on what other folks have had to 
say at the table. If there are any volunteers — Mr. Glazer. 

Mr. Glazer. Thank you, Mr. Chairman. You correctly focused in 
on the Senate, the legislation, and I think that is clearly the issue. 
Commissioner Kerr mentioned that there are provisions in the leg- 
islation, or being talked about — they are not actually in the legisla- 
tion — to delay FERC’s standard market design initiative. The Com- 
missioner is right. I mean, you can read the language lots of dif- 
ferent ways 

Senator Voinovich. Standard market 

Mr. Glazer [continuing]. Design 

Senator Voinovich [continuing]. Basically is the overall plan 
that looks at the entire transmission grid, looks it over and comes 
back with recommendations on how it can be improved and then 
tries to determine how individual companies, RTOs, States get 

Mr. Glazer. It is a plan to actually sort of set forth some stand- 
ards around the country. One of the big issues was we have seams 
around the country, and Ohio is a good place, a good example of 
that. And the idea behind what FERC was trying to do was saying, 
well, let us have some basic rules of the road. Let the markets look 
like this. This commodity doesn’t respect State lines. Let us have 
some basic rules with regard to markets, with regard to planning, 
reliability, etc. So it was a broad brush approach. 

Some may argue it was too much, too little. Personally, I am very 
concerned about a provision that would come down that would just 
delay things, because as I mentioned, delay is the kiss of death on 
Wall Street. 

Senator Voinovich. I think some people were saying delay 3 
years or something like that, and my personal feeling is that we 
have waited too long. 

Mr. Glazer. Exactly. 

Senator Voinovich. We are so long overdue on this that it is not 
time for us to delay and look and try to figure out where we are 
going. What people don’t understand is that this is a capital-inten- 
sive industry and people are not going to invest in something if 
there is uncertainty about what the future looks like. They are just 
not going to invest. It is the same thing with nuclear energy. One 
of the problems in terms of building new nuclear energy plants is 
what to do with the waste? That is why Yucca Mountain is very 
important. We finally decided we are going to go forward with this. 
So you are going to probably see some new nuclear plants in this 
country because investors know that that issue is taken care of 
long term. 

Mr. Glazer. And that is the problem with delay. I would rather, 
if the Federal Energy Commission comes out with something that 
the Congress of the United States thinks is inappropriate, you have 
the tools to change it. When I was on the PUCO, if the legislature 
didn’t like something PUCO did or you didn’t like something PUCO 
did or the Supreme Court, there were lots of checks and balances. 
But delay is just the kiss of death to investment. I would rather 
let the FERC move forward. If the Congress doesn’t like this provi- 
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sion or that provision, it certainly can weigh in. But delay is the 
kiss of death for the reasons you stated. 

Senator Voinovich. Mr. Schriber. 

Mr. Schriber. Mr. Chairman, just to underscore what Craig has 
told you, the one thing that Dr. Cooper said, and maybe the only 
thing that I really agreed with, is that Ohio should have a voice 
in the outcome of all this and you are our voice. As Craig has sug- 
gested, and I wholeheartedly agree, delay is not the way to go. If 
there are any provisions that would handcuff the FERC from mov- 
ing forward, I think it would be very unfortunate. 

Senator Voinovich. Any further comments? 

Mr. Cooper. Mr. Voinovich, I am not sure you want to be respon- 
sible when the lights go out and have them call you. Let us be 
clear. The Chairman of the 10 Commission hears about it. The fun- 
damental question here on the SMD was not a question of — there 
is very little in the SMD that had to do with reliability. The SMD 
sort of punted on that question. What the SMD has is an economic 
model for transacting transmission rights and electrons, right, and 
if the FERC hadn’t bothered with the transmission rights, it might 
have gotten away with its wholesale markets. 

But this was a model that two-thirds of the country — let us be 
clear. You have got Ohio. You have got New York. They have been 
here. But two-thirds of the States have not chosen their deregula- 
tory model, and in our view, the SMD was coercing the other 
States in the country through its market design requirements to 
pursue this path. 

So you need to decouple the deregulation issues from the reli- 
ability and the transmission issues. If you do that, you will have 
a lot more support for expanding and devoting more attention to 
the national highway system for electrons. 

Senator Voinovich. What you are basically saying is that there 
are some people that haven’t yet decided what they want to do and 
they shouldn’t be forced in it. Your opinion is that the standard 
Market Design would force them into it. Does anyone want to com- 
ment on that? Mr. Kerr. 

Mr. Kerr. It would absolutely, Mr. Chairman, as Chairman 
Wood said today, that in this rule, they have moved beyond Order 
2000 and said that you would mandatorily be required to join a re- 
gional transmission organization as part of the SMD. And again, 
I think we need to parse words here. Being a lawyer in my former 
life, I am guilty of that. Delay, I think a lot of mistakes are avoided 
by taking your time. So, I mean, we can comment generally about 
whether delay is good or bad. 

But as I read the language that came in Senator Domenici’s sub- 
stitute, it says only that FERC shall not issue a rule with general 
applicability related to the standard market design. What I have 
not heard, people have said, well, this could delay what we want 
to do in Ohio. I don’t see how that is possible, because that would 
not be a rule of general applicability if it were confined to a par- 
ticular RTO or ISO. 

If that is true, I think that language could be very carefully im- 
proved upon. You could put a “however” clause afterwards. You 
could say, however, nothing in this section is intended to stop Mr. 
Torgerson or Mr. Glazer or Dr. Schriber from moving forward in co- 
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operation with the Federal regulators to adopt the market rules 
that they want to apply within their region. 

So again, when I look at the language, I do not see the basis for 
this agreement, “don’t handcuff FERC.” In contrast, I know that if 
FERC goes forward with this rule, its position will be that the en- 
tire Southeast and the West and other regions of the country that 
are in different structures, that have very serious — who have stud- 
ied these proposals and have continued to have very serious con- 
cerns about whether this is correct for us would, in fact, be forced 
to go forward. 

So I don’t think it is that our region wants to stop Dr. Schriber’s 
region. We certainly don’t. In fact, we would help in any way we 
could. But I think the question ought to be asked, should your 
State, should Ohio force upon us what it needs to solve its prob- 
lems. I think clearly it should not be, and this language allows that 
because it says you can’t put a national plan out, but it doesn’t pro- 
hibit you from going forward and working maybe with instead of 
a hammer, a surgical scalpel to work on the various regions to im- 
prove upon the systems. 

They are doing a lot of good things in the Midwest. They are 
doing a lot of good things in the Northeast. And what we ought to 
do is take what is good and improve upon it in every region of the 
country, but certainly not go backwards in our regions of the coun- 
try. So with that, I 

Senator Voinovich. I would like to get one more comment in re- 
gard to what Mr. Kerr said, and I think that what I am going to 
do is wrap it up with one last comment on what you made ref- 
erence to, Mr. Kerr, and then we are going to adjourn the hearing. 

Mr. Museler. 

Mr. Museler. Thank you, Mr. Chairman. Just a few comments 
on that. The first is that there is some regionally from the stand- 
point of market design. I think that is factored into the SMD rules 
that FERC wants to promulgate. 

Senator Voinovich. So you don’t think the SMD rules are going 
to force people, as Mr. Kerr has suggested — or, no, Mr. Kerr. You 
believe the language is broad enough so that it doesn’t force you 
into 

Mr. Kerr. The language in the SMD would, in fact, make our 
participation in an RTO mandatory. The language — the point I was 
making is that the language in Senator Domenici’s substitute 
would allow all the regions to proceed as they chose to. So two dif- 
ferent documents. 

Mr. Museler. The point I would like to make is that the stand- 
ard in standard market design matters. The design of the markets 
matter. California is an example of what happens when you don’t 
get it right. I am not saying they all need to be the same, but they 
do need to be consistent and they do need to make economic sense, 
because whatever the region is, whatever regions choose to say 
they are the region, there are seams, as Mr. Glazer pointed out, 
and there needs to be consistency in those rules, both for reliability 
and for market operation. You cannot have too much diversity in 
those market rules or you will not have interstate commerce occur- 
ring the way it should. 

Senator Voinovich. Any other comments before I run? 
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Mr. Glazer. Just a quick one. I think FERC, we should give 
them some credit. They really did back down from sort of the more 
mandatory parts of their standard market design. The rule as it 
presently is being proposed has that regional flexibility. So I think 
that they have tried to make the balance between what Mr. 
Museler said and what Commissioner Kerr would say, so I 

Senator Voinovich. Is there anything that FERC can or should 
do with market design that has enough flexibility to work for all 
of you? 

Mr. Glazer. I think the white paper that — they just issued what 
is called a white paper. I think it provides that flexibility in there. 
They really did hear the message from the Congress. So I think 
that flexibility is in there. There is this issue about whether you 
mandate RTOs or not. 

The problem there isn’t, what if one region, what happens. What 
happens if one utility doesn’t want to play but all the utilities 
around it want to play? Then you have got a problem. You have 
got an electrical problem again. 

Senator Voinovich. There has got to be some provision that says 
if that kind of thing happens, that somebody is going to step in and 
make it happen. 

Mr. Glazer. Somebody has got to, right. 

Senator Voinovich. Absolutely. And I think the other thing that 
you need to look at is that we are today, tomorrow, 5 years from 
now, 10 years from now, and God only knows just how this thing 
is all going to work out, but more and more, we have electricity 
moving around and I am sure somebody smarter than I am can get 
into what happened in California. But it really appeared to me that 
somebody was not doing what they were supposed to be doing in 
terms of developing a grid so that that situation would not have 
occurred. 

Mr. Cooper. Mr. Chairman, let me offer one point about the 
seams question, and that gets to the fundamental proposition 
that — the desire to have a Federal backstop. You have heard it said 
that the States don’t do their job. I want you to do more than that, 
and here is what I want you to do. I want a formal process, and 
the Congress ought to take the responsibility for establishing a for- 
mal process of State compacts or some other mechanism so that it 
is not simply a question of whether one State disagreed or not, but 
a process has to be set up by which the States can sit together and 
reconcile their differences. 

If you look at what has happened in the Midwest, people have 
jumped in and out. The industry members have jumped in and out. 
I think the State officials, if they were sitting together with the au- 
thority to make that decision, would have done a much better job 
than the industry has jumping in and out. 

So it is a governmental responsibility to make interstate com- 
merce flow effectively, and that is not in anything before the Con- 
gress. 

Senator Voinovich. This has been an interesting panel. I apolo- 
gize for the interruptions that we have had. Thank you very much. 

Again, thank you for your attention and your courtesy in being 
here, and this hearing is adjourned. 

[Whereupon, at 11:32 a.m., the Committee was adjourned.] 




KEEPING THE LIGHTS ON: THE FEDERAL 
ROLE IN MANAGING THE NATION’S ELEC- 
TRICITY 


THURSDAY, NOVEMBER 20, 2003 

U.S. Senate, 

Oversight of Government Management, the Federal 

Workforce, and the District of Columbia Subcommittee, 
of the Committee on Governmental Affairs, 

Washington, DC. 

The Subcommittee met, pursuant to notice, at 10:07 a.m., in 
room SD-342 Dirksen Senate Office Building, Hon. George V. 
Voinovich, Chairman of the Subcommittee, presiding. 

Present: Senators Voinovich, Carper, and Lautenberg. 

OPENING STATEMENT OF SENATOR VOINOVICH 

Senator Voinovich. Good morning. As one who has hearing aids, 
I can understand your problem. 

First of all, I would like to say that I am glad that I am Chair- 
man of the Subcommittee on the Oversight of Government Manage- 
ment, the Federal Workforce and the District of Columbia, because 
this hearing could probably be held in the Environment and Public 
Works Committee, of which I am also a member. But having the 
chairmanship of this Subcommittee gives me some authority to 
oversee different areas of government, and I thought it was very 
important that we deal with this subject before this Subcommittee 
on Oversight of Government Management for its significance in 
terms of the issue as to the public’s relying on electricity and also 
because it is an important issue in the State of Ohio, where a lot 
of this occurred. 

This is the second hearing that we have held on the blackout 
that hit the Midwest and the Northeast on August 14, and the 
proper Federal role on managing our electricity system. It is now 
well-documented that the August 14 blackout was the largest 
blackout in our Nation’s history. Over 50 million people lost power 
that day, including over 2 million people in Ohio. 

What has been lost in the shuffle here in Washington, however, 
is the impact that the event had on the economies of the Midwest 
and the Northeast. The Ohio Manufacturing Association estimates 
that this blackout directly cost Ohio manufacturers over $1 billion, 
a huge hit that could not have come at a worse time, given that 
millions of American manufacturing jobs are already at risk. 

It is absolutely imperative that we do all we can to prevent such 
events from happening in the future. As I mentioned at our first 

( 37 ) 
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hearing on this topic, our Nation is currently served by an overbur- 
dened and heavily strained electricity system that was not de- 
signed for the widespread wholesale transactions that currently 
take up a large part of its capacity. 

Over the last several decades, our transmission capacity has 
lagged behind both generation and demand increases. We must 
take concrete steps now to strengthen our grid by establishing reli- 
ability standards that are mandatory and enforceable. We need 
new investment in transmission capacity, and we need to strength- 
en existing Regional Transmission Organizations so that we can ef- 
fectively manage the grid to prevent future blackouts. 

A lot has happened since we held the first hearing back in Sep- 
tember. First, the U.S.-Canada Power System Outage Task Force 
that was established to investigate the blackout has issued an in- 
terim report entitled, “Causes of the August 14 Blackout in the 
United States and Canada.” Second, a House-Senate conference 
has reported a comprehensive energy bill that contains electricity 
provisions which will significantly affect the management of our 
national electricity system. It is now pending business here in the 
U.S. Senate, and I am prayerful that it is not filibustered so that 
we cannot move forward and get it done before we go home. 

I want to commend the administration for its leadership on elec- 
tricity transmission issues and the August 14 blackout. President 
Bush has moved quickly to create the U.S.-Canada Joint Task 
Force on the Power Outage, and I appreciate the fact that the Ca- 
nadians have cooperated and strongly pushed for a more reliable 
electricity grid in order to prevent future blackouts. 

Secretary Abraham has overseen significant changes in the util- 
ity sector over the last 3 years — I had an opportunity to talk to him 
about that yesterday when I saw him — during which time there 
were two major blackouts. He issued an important study on the 
transmission grid and created a new Office of Electricity Trans- 
mission and Distribution at the Department of Energy. Simply put, 
the administration has made our national electricity system a na- 
tional priority — as it should be. 

I would also like to comment on the electricity title in the con- 
ference report on H.R. 6. Following the August 14 blackout, I, along 
with several of my colleagues, called on the energy bill conferees 
to include provisions that would help prevent future blackouts in 
the conference report. I also asked the witnesses at our first hear- 
ing, which was on September 10; Mr. Wood, you were here for that; 
what they thought we needed to do legislatively in order to prevent 
future blackouts. Chairman Wood, you probably remember me ask- 
ing you that question. The response from the witnesses, including 
Chairman Wood, was that the best legislative fix would be to enact 
electricity provisions in the comprehensive energy bill including 
mandatory reliability provisions, provisions to increase investment 
in the transmission grid, and provisions to grant Federal siting au- 
thority to FERC. 

The energy bill conferees obviously listened. The electricity title 
to the conference report will, when enacted, establish mandatory 
reliability standards that will be implemented and enforced by 
FERC. It will encourage new investment in the transmission grid. 
There is a lot of money in there to do that. It will grant Federal 
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siting authority to FERC. And, although it delays implementation 
of the Standard Market Design rulemaking — we talked about that 
again at our last hearing — it will allow FERC to strengthen exist- 
ing Regional Transmission Organizations in order to ensure that 
problems and mistakes — like the ones detailed in the interim re- 
port we are discussing today — are eliminated in the future. 

The House passed the conference report by a bipartisan vote of 
248 to 160 on Tuesday, and we will be voting on cloture tomorrow 
morning on this bill. The Senate needs to follow suit and send this 
critical legislation to the President as soon as possible — it is very 
important we this done. 

As I mentioned earlier, the purpose of today’s hearing is to dis- 
cuss the interim report entitled, “Causes of the August 14 Blackout 
in the United States and Canada,” that was issued yesterday by 
the U.S.-Canada Power System Outage Task Force. Before we pro- 
ceed, I would like to include the interim report in the record. With- 
out objection, it is ordered. 1 

I understand that the administration is currently planning to ac- 
cept public comments on this interim report and then publishing 
a final report early next year. I intend to hold a final hearing on 
this topic when the administration releases the final draft and 
makes its recommendations as to what further steps need to be 
taken to prevent such an occurrence from happening again, and I 
would be really interested if any of you witnesses want to comment 
about whether this conference report contains enough to get the job 
done? And if you do not believe that it does I would like to know 
what your ideas are on what other things we need to have in order 
to give you the tools to get the job done. 

We have got an impressive lineup of witnesses this morning to 
outline the preliminary findings of the task force. I look forward to 
an informative discussion. 

Our first witness today is the Hon. Pat Wood, the Chairman of 
the Federal Energy Regulatory Commission. And joining him on 
behalf of the administration is James Glotfelty, the Director of the 
Office of Electricity Transmission and Distribution at the Depart- 
ment of Energy, a new job, and Michehl Gent is the President and 
CEO of the North American Electric Reliability Council, and I 
think that the acronym is NERC. We are very happy to have all 
of you here today, and thank you for testifying. 

Gentlemen, it is the custom of this Subcommittee that we swear 
in our witnesses, and I wish that you would rise, and I would ad- 
minister the oath to you. 

[Witnesses sworn.] 

Senator Voinovich. Let the record show that the witnesses an- 
swered in the affirmative. 

Mr. Wood, we will start with your testimony. 


1 The report entitled “Interim Report: Causes of the August 14 Blackout in the United States 
and Canada” appears in the Appendix on page 211. 
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TESTIMONY OF HON. PAT WOOD, III , 1 CHAIRMAN, FEDERAL 
ENERGY REGULATORY COMMISSION 

Mr. Wood. Good morning, Mr. Chairman. It is nice to be back. 
I appreciate the opportunity to discuss the very heavy report yes- 
terday that was presented by Secretary Abraham and Minister 
Dhaliwal. 

Watching and studying the blackout for me, and I think for a 
number of us, has been a sobering experience. The reliability of the 
North American electric system is normally so excellent that this 
year’s notable service interruptions from the August 14 blackout 
here in the Northeast, blackouts overseas in London, Italy, Argen- 
tina, Scandinavia, and also and recently back here again from Hur- 
ricane Isabel and related weather damages have forced us all to 
look afresh at all of the old assumptions that we have about the 
value of reliable electric service and what it takes to keep the 
lights on. 

So here are some thoughts from an energy regulator about what 
I have learned from this blackout investigation, from the thorough 
investigation encompassed in the interim report, and from thinking 
about these other blackouts that we have seen in the past year. 
The blackouts in the Northeast, in Italy, and in London and else- 
where, have a common theme: Something routine happens, like a 
tree contacting a power line, or a minor relay setting trips because 
it was done wrong, and the time to react and keep the system sta- 
ble suddenly shrinks beyond the capability of human control, when 
the machines take over. 

The grid is a tremendously complex machine, and the inter- 
connectedness that allows us to benefit from both higher reliability 
and lower costs in all hours also causes the domino failures experi- 
enced in many countries in recent months. We cannot ever prevent 
blackouts, but we can and must learn to reduce their frequency, 
magnitude and impact. 

The best way to manage blackouts is to prevent them, not to 
hope for heroic rescues when we are already in a jam. And the se- 
cret to reliability lies in making sure that every transmission 
owner, control area operator, and reliability coordinators takes care 
of the basics: Adequate tree trimming, adequate training for emer- 
gency as well as routine operations, effective communications with- 
in and across organizations, and having effective backup facilities, 
procedures and tools. 

The investigation clearly shows that had First Energy trimmed 
its trees, used a solid state estimator program after the trip of the 
East Lake 5 Unit along the lake, and regularly throughout the 
afternoon of August 14, and trained its operators to better recog- 
nize and deal with these emergencies, the blackout would not have 
happened. The blackout study shows that the current reliability 
standards were violated by First Energy and the Midwest Inde- 
pendent System Operator. We need better compliance and tough, 
cl63i* st 3 .nd 3 .rds 

The FERC will be working closely with NERC and the stake- 
holders to develop those standards and to implement the reliability 
provisions of the energy bill if Congress approves it. 


lr The prepared statement of Mr. Wood appears in the Appendix on page 193. 
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In anticipation of approval and because the timelines are so 
short, and the needs are so great, on December 1, the Commission 
has scheduled a conference to discuss the implementation of the re- 
liability provisions in the statute, in order that we can have man- 
datory rules in place and operational by this summer. We do need 
some major investments in new transmission facilities and new 
grid technologies, especially those that make it easier for us to 
manage the basics of electricity. But we need to make these invest- 
ments wisely, for lines and equipment that expand the reliability 
parameters of the grid where it is needed, for example, new sources 
of reactive power in the Cleveland-Akron area. These appear to be 
long overdue. 

Further analysis conducted by the blackout investigation teams 
will teach us much about how the cascade spread, why it stopped 
where it did, and those things will help us to design a system that, 
over the long-term, should perform more reliably and cascade more 
narrowly. The new energy bill offers options to site long-needed 
transmission lines and to pay for the reliability investments, and 
I am eager, as my colleagues are at the Commission, to put these 
measures into place. 

We also need to invest in hardware and software that let opera- 
tors manage the grid more effectively. Tools that improve system 
monitoring, evaluation, visibility, and information sharing about 
the grid operations over a wide region will allow operators to man- 
age the grid more reliably on a day-to-day basis as well as in emer- 
gencies. Our colleagues at the Department of Energy have done 
some excellent work in this area over the past few years, and we 
will be looking to these technologies and others to raise the bar for 
effective grid management. 

Transmission is regulated at both the Federal and the State 
level. Clearly, we need to regulate it better to assure that the reli- 
ability that Americans have come to expect is, in fact, delivered. As 
the present energy bill recognizes, the days of voluntary reliability 
standards with no enforcement teeth must end. Federal regulators 
must work closely with our State colleagues to make sure that util- 
ity cost-cutting that allows 14-inch diameter trees to grow in a 
transmission right-of-way or inadequate operator training or the 
widespread use of inadequate software ineffectively used must end. 

I pledge that my Commission will work closely with our col- 
leagues in Ohio and other States to deliver better regulation for 
better reliability. I do note with that that my former panelist, Mr. 
Shriver, from the State of Ohio, announced yesterday some reme- 
dial measured that they have initiated with the utilities in Ohio al- 
ready, with the governor’s support. 

Some claim that electric competition and higher energy flows 
caused underinvestment in an overworked grid and made this 
blackout inevitable. What they ignore is that the operators’ pri- 
mary challenge is to work the system that you have and that the 
operator has the power to cut back any transaction, whether it is 
a long distance transaction or one to serve local load; to tighten the 
operational limits on any transmission line or power plant; and 
even to cut customer load if that is what it takes to keep the sys- 
tem safe and secure. Markets do not compromise reliability, but we 
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must redouble our efforts to assure that all necessary reliability 
measures are taken. 

Perhaps the saddest portion of the blackout report is Chapter 6, 
the comparison of this outage to those that have happened since 
1965. The common factors are overwhelming: Conductor contact 
with trees due to inadequate vegetation management; insufficient 
reactive power; inability of system operators or coordinators to rec- 
ognize and understand events across the broad, regional system; 
failure to ensure that system operation was within safe limits; lack 
of coordination on system protection; failure to identify emergency 
conditions; ineffective communication; lack of safety nets; and inad- 
equate personnel training. 

The seven outages that the report reviewed from 1965 to 1999 
include all of these elements. Extensive analysis followed each out- 
age, and blue ribbon panels were developed with good recommenda- 
tions after each of these outages. Some of the recommendations 
that have followed these outages have been implemented but not 
many. It is my hope that with the adoption of the new reliability 
provisions in the energy bill, we can finally implement most if not 
every one of these recommendations and stop repeating the mis- 
takes again and again. The cost of the mistakes is high for our Na- 
tion, for our sister nation to the north, and all of our citizens de- 
serve better. 

The cost of blackouts is immense, both in human and financial 
costs. I have seen estimates every day that try to impact the cost 
not only to your home State, Senator, but to the entire North- 
eastern quadrant of the continent. New transmission facilities and 
tools are not cheap, and business practices are not cheap, either, 
but these improved business practices will need to be paid for; they 
will need to be part of the overall cost of electricity, and as an eco- 
nomic regulator, I am prepared to put those in the rates and justify 
that is in the public interest. 

But if you ask New Yorkers who were stuck in the Subway on 
August 14 or the Cleveland residents who had to boil their water 
for days or folks around Maryland and Virginia who sat without 
power for as long as a week after Hurricane Isabel, most would tell 
you that they would rather pay a little more for a reliable electric 
system than reduce their bills to avoid incremental, reliability and 
improving costs. 

So I do think it is important to recognize that there will be a cost 
to improve this system. I think that it is one that, as a Nation, we 
should pay, because the benefits far outweigh the costs. But again, 
as an economic regulator, I do think the findings here were very 
important and force us all to rethink the paradigm that we have 
been operating under. 

I look forward to your questions, Senator Voinovich and Senator 
Lautenberg. Thank you. 

Senator Voinovich. Thank you, Mr. Wood. Mr. Glotfelty. 
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TESTIMONY OF JAMES W. GLOTFELTY , 1 DIRECTOR, OFFICE OF 

ELECTRIC TRANSMISSION AND DISTRIBUTION, U.S. DEPART- 
MENT OF ENERGY 

Mr. Glotfelty. Thank you, Mr. Chairman and Senator Lauten- 
berg. 

I appreciate the opportunity to testify before you today and out- 
line the findings of the U.S. -Canada Power System Outage Task 
Force investigating the blackout that occurred on August 14. 

Three months ago today, large sections of the United States and 
Canada were still recovering from one of the largest power black- 
outs in our Nation’s history. Since that blackout, hundreds, lit- 
erally hundreds, of technical experts have worked tirelessly, long 
hours, sleepless nights, to help the U.S. -Canada Task Force deter- 
mine how and why this blackout occurred. 

Yesterday, as you know, the task force released the interim re- 
port that marks our progress to date in the search for answers on 
what happened that day. The interim report focuses on events, ac- 
tions, failures, and conditions that led to the blackout and caused 
it to cascade over such a large region as well as questions relating 
to the nuclear power plant operations in both countries and the se- 
curity of our grid control systems. It presents facts collected by the 
investigation team, and it does not offer speculative or unconfirmed 
information or hypotheses. 

Without going through a line-by-line review of how the system 
failed, I would like to walk you through the causes that we outlined 
in the interim report. But before I do this, I would like to make 
it clear that it is the control area operator, in this case, First En- 
ergy, who had the primary responsibility to maintain system reli- 
ability, regardless of the conditions. They are required to have the 
tools to ensure that the grid is reliable. They must be able to take 
all actions necessary to ensure a reliable system. 

With that caveat in mind, I will walk you through the causes 
that we outlined in our report. The first type of cause: First Energy 
did not properly assess the changing conditions on their system. 
They did not use an effective contingency analysis tool routinely. 
They lost their monitoring alarm systems and lacked procedures to 
understand that. After they made repairs, they did not check to see 
if they were effectively working to monitor the system. And once 
both systems failed, they did not have effective backup tools to en- 
sure that they really had a basic understanding of the system con- 
ditions before them. 

Second, First Energy failed to adequately maintain its trans- 
mission rights of way. This seems so very basic, yet, as Mr. Wood 
said most of the blackouts that have occurred in this country and 
overseas, some portion of that or some cause of that deals with in- 
adequate vegetation management in our rights of way. Our report 
specifically stated: Overgrown trees in First Energy’s transmission 
rights of way caused the first three major 345 line failures in Ohio. 
These lines trip when contacting trees that had grown past their 
maximum allowable limits in their rights of way. 

Our investigators found that First Energy rights of way being 
clean are not a new problem. They found one tree over 42 feet tall 


lr The prepared statement of Mr. Glotfelty appears in the Appendix on page 196. 
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in a right of way that they approximated the age was 14 years old. 
Another was 14 inches in diameter currently in the rights of way. 
These trees do not grow overnight. This means that there is a long, 
systemic issue that needs to be dealt with not only with First En- 
ergy, but it needs to be looked at by utilities all across the country 
on how we ensure our rights of way are maintained. 

It seems so very basic that we would maintain our rights of way. 
However, it does conflict reliability of our system grid with land 
owner rights, and that is something that the State commissions as 
well as FERC will have to deal with in the coming months. 

The third and final group of causes of this blackout deal with re- 
liability coordinators, in this case, the Midwest ISO. They were un- 
able to provide adequate diagnostic support over the entire region 
to help First Energy respond to their problems. Their State esti- 
mator failed. Their monitoring equipment did not have real-time 
line status and information. Their operators could not identify 
where lines had tripped. And the Midwest ISO and their neighbor, 
PJM, did not have adequate measures to understand issues on the 
seams between the two borders. 

According to NERC and outlined in our report, these failures 
amount to at least six NERC reliability standards being violated, 
four by First Energy and two by MISO. Hopefully, the Congress 
will take action on the energy bill and make these rules manda- 
tory, and we can move down the road to ensuring that we have stiff 
penalties for violation of reliability rules. 

Mr. Chairman, I would like to reference a critical point in this 
investigation: 3:05 in the afternoon on August 14 is the critical 
time frame. At that time, the investigation’s extensive modeling de- 
termined that the system was being operated reliably, within safe 
operating limits. That fact alone eliminates a number of possibili- 
ties as causes of the blackout. It eliminates high power flows to 
Canada, of which the majority of the power flows going across First 
Energy’s system were actually ending in First Energy’s system. Ap- 
proximately 20 percent of First Energy’s load was being imported. 

System frequency variations; low voltages earlier in the day and 
prior days; low reactive power output from independent power pro- 
ducers; outages of individual generators and transmission lines 
that occurred well in advance of the blackout; all of those were con- 
sidered by the investigations team, modeled and discarded as not 
causes of the blackout. 

Finally, the task force spent time understanding the nuclear 
plants and the security of the system. The report outlines that all 
of the nuclear plants in the United States and Canada shut down 
safely. They were not a cause of the blackout. They were reacting 
to system conditions and tripped themselves from the grid. The se- 
curity group found that there was no terrorism or deliberate cause. 
There were no SCADA system violations with the information that 
they have reviewed to date and no computer viruses that caused 
any of this blackout. 

Phase one of our task force investigation and the public’s re- 
sponse to it will give us a wealth of information that will be the 
basis for forming recommendations. Phase two of our investigation 
will include three public forums in Cleveland, New York City, and 
Toronto in early December. These forums will offer an opportunity 
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to all of those listed in this report as well as other interested par- 
ties to provide the task force with comments and recommendations. 
The task force will then issue a final report containing our rec- 
ommendations for improving the electric system and its reliability. 

Thank you, Mr. Chairman. I would be happy to answer any ques- 
tions. 

Senator Voinovich. Thank you, Mr. Glotfelty. Mr. Gent. 

TESTIMONY OF MICHEHL R. GENT , 1 PRESIDENT AND CHIEF 

EXECUTIVE OFFICER, NORTH AMERICAN ELECTRIC RELI- 
ABILITY COUNCIL 

Mr. Gent. Thank you, Mr. Chairman and thank you, Senator 
Lautenberg. You probably do not know this, but you are my Sen- 
ator in the State of New Jersey. I appreciate your being here this 
morning. 

Senator Lautenberg. I am glad to represent you. It depends on 
your testimony. [Laughter.] 

Mr. Gent. Thank you all for inviting me this morning to speak 
to NERC’s perspective on the interim report of the U.S. -Canada 
Power System Outage Task Force on the causes of the blackout. 
NERC, as most of you know, is a not-for-profit organization that 
was formed after the Northeast blackout in 1965. Our job is basi- 
cally to prevent blackouts from happening. That cascading outage 
of 1965 was supposed to have been the last one, and it was not. 
So this study adequately covers what the problems were leading up 
to this recent blackout. 

We are structured as a regional organization that every electric 
utility and member that participates in the electric system market 
belongs in one of 10 regional reliability councils. They own a not- 
for-profit organization, which is NERC. NERC has been an integral 
part of the joint fact-finding investigation that led to the interim 
report that was issued yesterday. We fully support the report’s 
findings and conclusions, and I would like to add that I fully sup- 
port the testimony of Mr. Glotfelty and Mr. Wood here this morn- 
ing. 

With respect to what happened, the key findings and conclusions 
may be difficult to find, but I will reference page 23 for the infor- 
mation that Mr. Glotfelty briefly described, and on page 25, you 
will see the NERC standards that we believe that we have deter- 
mined have been violated. 

Immediately after the onset of the blackout, NERC began assem- 
bling a team of the best technical experts in North America to in- 
vestigate exactly what had happened and why. Every human and 
data resource that we have requested of the industry has been pro- 
vided, and experts covering every aspect of the problem have volun- 
teered from across the United States and Canada. 

In the week following the blackout, NERC joined with represent- 
atives of DOE and the Federal Energy Regulatory Commission to 
establish a single joint fact-finding investigative team. The ques- 
tion has often been asked: Are there more than one investigation 
underway? And the answer is no. We stand side-by-side in this. 


1 The prepared statement of Mr. Gent with attachments appears in the Appendix on page 200. 
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All of the members of the team, regardless of their affiliation, 
have worked to help correlate and understand the massive amount 
of data that we have received. We have hundreds of volunteers 
from organizations all over North America, and we believe more 
are to come as we venture further into the investigation. 

To lead our NERC effort, we established a steering group of the 
industry’s best executive-level experts from systems not directly in- 
volved in the cascading grid failure. The steering group’s scope and 
members of that group are described in our Attachment A to our 
written testimony. 

On October 15, I sent a letter to the CEOs of 160 control areas 
and reliability coordinators across North America that control our 
electric grids, and I directed them to verify that within 60 days 
that their organizations are measuring up to reliability require- 
ments in six key areas. Those are also described in an attachment 
to my written testimony. Those responses are due on December 15. 
The purpose was to make sure that we are reducing, to the extent 
possible, the likelihood of any further action like the blackout. 

Chapter 6 of the interim report has been mentioned this morning 
by Mr. Wood. It compares this blackout to blackouts that we have 
seen in the past, and while it is true that the same things seem 
to continue to crop up as the reasons, it is also true that we have 
a number of situations where automobiles were involved in deaths, 
and we have not been able to stop that. I do not mean to be flip 
about this, but the area of study is so wide that we are virtually 
unable to totally prevent these things from happening. 

What we have done, though, is we have made tremendous 
strides. The whole reliability coordinator system is a result of rec- 
ommendations of a previous blackout. We now certify our opera- 
tors. That is the result of the recommendations of the report. And 
we have taken other large steps. 

One important step that Congress can take is, as you have indi- 
cated earlier, Mr. Chairman, is to pass the reliability legislation or 
the energy legislation with the reliability language in it. I believe, 
as you asked, I believe that legislation, the reliability part, is ade- 
quate. It will provide us with the assurances that we need to see 
that the rules are developed correctly and that they are enforced 
and complied with. 

As for the August 14 outage, much remains to be done. As the 
entity responsible for reliability standards, NERC must understand 
and communicate with its members what happened on August 14 
and why. The interim report is a major step; in fact, it may go be- 
yond a major step. This may be the finest document of its type ever 
produced, even though it is the result of a disaster. We must also 
determine if there are other standards that have been violated. We 
must determine if our standards are adequate. We must make 
modifications to take into account what happened with this black- 
out and how the system is now being used. 

We will continue to work with the task force. The investigation 
will proceed, and recommendations will be developed. We expect to 
learn many more lessons from this event, and I expect that I will 
be back here again some day in the future. 

Thank you, and I would be happy to take your questions. 

Senator Voinovich. Thank you, Mr. Gent. 
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We have been joined by Senator Lautenberg. Senator Lauten- 
berg, would you like to make a statement before I start the first 
round of questions. 

OPENING STATEMENT OF SENATOR LAUTENBERG 

Senator Lautenberg. I would appreciate the chance to just make 
a short statement. I would like to first welcome Mr. Gent. 

And I thank you, Mr. Chairman, for convening this second hear- 
ing on the electricity blackout. It takes someone who has been a 
governor and a mayor, who has been up front with the problem, to 
recognize the importance of getting on with this thing and not let- 
ting it linger, and we appreciate your direction and your action 
here. 

The critical loss of power on August 14 brought a large part of 
the country to a standstill, and we still have unanswered questions, 
as all three of you have identified. One of the questions that arises 
for me is heaven forbid that terrorist organizations that we know 
threaten us from many, many points and could coordinate some- 
thing with the lights out would be devastating in terms of not only 
the damage that might occur but the panic that would follow if 
news ever got out that there was something underway that was at- 
tacking the American people in that area. 

This event dramatically demonstrated our vulnerabilities in the 
Nation’s electrical grid and the need for mandatory reliability 
standards. Now, if we fail to correct the flaws in the Nation’s elec- 
tricity transmission system, experts, they say that other parts of 
the country will suffer similar blackouts. I think that is a given at 
this point. Blackouts come with a high price tag: Massive public in- 
convenience, increased danger for citizens who find themselves in 
the dark. Reliable electricity is not a new issue. 

Some regions have made great progress, while others remain 
locked in outmoded systems dating back to the beginning of elec- 
tricity regulation in this country. And I understand that some of 
my colleagues have concerns about deregulation of the electric in- 
dustry, but I would like for them to take a look at what has hap- 
pened in New Jersey, where we are part of PJM, the Pennsylvania, 
New Jersey, Maryland interconnection, the country’s first fully-op- 
erating regional transmission, RTO, and the world’s largest com- 
petitive wholesale electricity market. 

And as an aside, Mr. Chairman, I am going to start a society to 
get rid of acronyms. [Laughter.] 

Because by the time we get finished with FERC, NERC, MISMA, 
MISO, and all of the other things, I do not know whether it is a 
Japanese menu that I am ordering from. [Laughter.] 

And they all get an explanation. So why bother trying to shorten 
them when we are going to lengthen them by a second statement 
of understanding? 

The ratepayers of New Jersey reaped enormous benefits from be- 
longing to this RTO, including stable rates that have not risen ef- 
fectively in 11 years. More to the point of this hearing: During the 
blackout, only 7 percent of PJM’s 25 million customers lost their 
power. Well, it is still a very significant number, but it is a long 
way from having the 100 percent blackout. And today, that is re- 
ferred from the witnesses, we want to talk about the need for more 
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RTOs like PJM, and given the multiple interconnections that exist 
across the grid, it strikes me that we need some kind of a regu- 
latory structure for regional, not just State or local transmission 
systems. I do not think that in that forum, it can be handled just 
by one State or by one community. 

So I welcome the release of the U.S. -Canada Task Force Interim 
Report, and hopefully, it will shine the light — we have not had a 
chance to examine it yet — but it will shine a light on the events 
and conditions which led to the blackout. And, of course, because 
the report was so recently received, we are going to need a little 
bit of time to fully digest the findings that it contains. And I do 
not know whether our witnesses have had the chance to read all 
of the words or every word in it, but we have experienced people, 
Mr. Chairman, good people who work on these things, and we con- 
gratulate you each for your part in that. 

And I have talked, the last time I mentioned this, to some of the 
employees at First Energy who came in to see me, and they com- 
plained bitterly about the antiquated state of transmission lines at 
First Energy. And this was not intended to be a labor dispute. We 
are not taking sides. But when the people who have to do the work 
say hey, this facility is outmoded, you ought to pay attention. And 
so, we did better than we might have, but when we look at the 
source of the problem, as it seems to be indicated, the source of the 
problem was where these folks were pointing when we had our con- 
versation. 

So, Mr. Chairman, I thank you for the courtesy of letting me 
make the statement, and I would be happy, after you, to ask some 
questions. 

Senator Voinovich. Thank you, Senator Lautenberg. 

I guess the new organization decided to get rid of that. [Laugh- 
ter.] 

I would like to congratulate all of you for your testimony this 
morning and also to underscore the fact that instead of everyone 
working individually on this investigation, that you have pooled 
your resources, and we were able to get Canada and the United 
States to work together on this issue. 

As many of you know, there were several other organizations 
that have taken a look into the blackout and have published re- 
ports on their findings, such as the State of Michigan, the Electric 
Power Research Institute, and the National Commission on Energy 
Policy, and I would like to enter these studies into the record with- 
out objection 

One of the things that I support in the energy bill is 
themandatory reliability standards with penalties. I also support 
the incentives for utilities to encourage investment in transmission 
lines, transmission lines are less of a payback than investing in 
generating electricity. And, of course, the other problem was envi- 
ronmental concerns and not in your own back yard kind of thing. 

That is all in the energy bill, and I think was it you, Mr. Wood, 
who said that you thought we could really move and review and 
come up with some new mandatory reliability standards, by when? 

Mr. Wood. I would like next summer, if the bill passes. We have 
180 days to adopt a rule to set up an electric reliability organiza- 
tion, which could be anybody but could be NERC, and at that point, 
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NERC and anybody else would come in and file to be declared. It 
is my intention that we basically say when you file, you also file 
the basic reliability rules. There will probably be some that are 
controversial that will take time to kind of hash out, but the entity 
that would be approved here, say, NERC, would design the rules; 
through their normal, open, transparent process. 

Senator Voinovich. But the point I am making is you are talk- 
ing about this summer. 

Mr. Wood. Of 2004. 

Senator Voinovich. But that is based on the fact that this legis- 
lation would be passed now. 

Mr. Wood. That is correct. 

Senator Voinovich. If it is not passed now, that means we are 
in limbo until such legislation is passed or this title is pulled out 
and considered as a separate piece of legislation. Most of us would 
not want to do that, but it could be necessary if this thing just con- 
tinues to be in limbo. 

Mr. Wood. Yes, sir. We are assuming, the fact I gave you would 
be an enactment in the next couple weeks. We would be able to go 
forward and do the proper open process that the law would require 
us to do. 

Senator Voinovich. Second, we talked — you were mentioning 
about Mr. Schreiber in Ohio heading up the PUCO, the Public Util- 
ities Commission of Ohio. 

Mr. Wood. Yes, sir. 

Senator Voinovich. You do not necessarily have to do this today, 
but there are different jurisdictions. In other words, if we pass this 
legislation, and FERC has a much larger role to play, what respon- 
sibilities would the PUCO have in Ohio and other public utilities 
commissions in other States have. Would they be responsible tree 
trimming, or would FERC or the organization that FERC would 
empower to do this? 

What is the difference in terms of jurisdictions between FERC, 
under the new legislation, and States? 

Mr. Wood. Under the new legislation, FERC would approve the 
standard that NERC would say this applies to either the whole 
country or to the eastern part of the country, what have you. This 
is how often you must do the tree trimming, and this is how much 
clearance you need to give it, etc. This is how you should train op- 
erators, backup plans; all the operational. 

NERC is also in charge, at the first instance, with enforcing com- 
pliance with those regulations, NERC or the ERO certified under 
the law. If someone complains that they were unfairly treated 

Senator Voinovich. Now, we want to make it clear that FERC 
is not anticipated to do this. You would authorize an organization 
like NERC or some other organization to go and be involved in 
this. Is that correct? 

Mr. Wood. Correct, and then, we would be, for example, a court 
of appeals if someone wants to contest the finding of NERC that 
they violated a rule, but by and large, that would be handled, at 
the first instance, by NERC and would not involve either the State 
or the Federal commission until an appeal is brought before us. So 
that would actually streamline, I think, the compliance process. Al- 
though we could end up handling a lot of appeals; the PUCO would 



50 


have independent authority under their State law. For example, 
when I was a Texas regulator, we had independent reliability au- 
thority over transmission and over distribution. From my under- 
standing, from what Mr. Schreiber said yesterday, they have clear 
authority over the distribution side, and they do not over the trans- 
mission side. 

Senator Voinovich. That protocol would have to be worked out, 
because you have different situations in different States, which I 
think it would be very important to understand that. 

Now, let us talk about transmission lines. It is my understanding 
that in terms of transmission lines and where they go that that is 
a State responsibility. 

Mr. Wood. Correct, yes, sir. 

Senator Voinovich. They are responsible for siting it, correct? 

Mr. Wood. Correct. 

Senator Voinovich. Now, what if the mandatory reliability orga- 
nization says that we need a transmission line in order to improve 
upon the grid, and the States say we are not going to do it? What 
happens then? 

Mr. Wood. The reliability provisions specifically state that the 
reliability organization cannot mandate construction of a trans- 
mission line or a generation plant. However, another provision in 
the law, in the proposed law before the Senate today, would em- 
power the Department of Energy, Mr. Glotfelty’s group, to identify 
national interest lines of a national nature, both large lines and 
multi-state lines. One year after identifying those lines, if a permit- 
ting process has not been successfully pursued and a permit re- 
ceived by a utility from a given State or States, then that would 
elevate up to the Federal Government to look at that. And we call 
it the backstop siting authority. 

So the States are still in the driver’s seat. It is only when they 
cannot act, or they are prohibited by their law from acting, or they 
choose not to act, that it comes to the Federal Government, in 
which case, we have to look at the issues you laid out, which are 
landowner, environmental siting. We might say no as well, but it 
is looked at on a broader scale and looked at on behalf of what is 
a national interest line. 

Senator Voinovich. So from a practical point of view, the reli- 
ability standard organization, say, NERC, if they get the job, would 
say this is really needed. They tell the States about it, and if the 
States refuse to act, if this was something that they considered to 
be very essential to the grid, would make that information avail- 
able to the Department of Energy 

Mr. Glotfelty. Correct. 

Senator Voinovich [continuing]. Who would then review the sit- 
uation, and a year after that, would come back and say yes, this 
is absolutely necessary and DOE would be able to take action to 
get the transmission line constructed; is that right? 

Mr. Glotfelty. That is possible; we 

Senator Lautenberg. May I interrupt just for a minute? I have 
got to go to another committee, but the record will be kept open 
so that we can submit questions, I assume, and I would ask our 
witnesses to respond as quickly as you can. 

Senator Voinovich. Thank you, Senator Lautenberg. 
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Senator Lautenberg. Thank you very much. 

Mr. Glotfelty. The process that we would like to go through for 
identifying national interest transmission corridors is not in place 
yet. Obviously, that is something that we need to do if the legisla- 
tion is passed. But we would not wait for NERC or the reliability 
organization to come to us with information for necessary, needed 
upgrades. We would do our independent modeling. We would work 
with the regions to determine national security, economic security 
or reliability lines that are necessary, and we would designate 
those in our own process. 

It will be a public and open process where everybody can partici- 
pate. Then, the State will — if the utility agrees and would like to 
build the line, then, that is when the State process gets triggered. 
And if they are not completed in a year, the authority would then 
go to FERC. 

Senator Voinovich. OK; so the fact is that we are putting incen- 
tives in here for companies to go ahead and invest in transmission 
lines. The State says fine; we will site this, but the utility says hey, 
we are not interested in going forward with it, what authority 
would you have in that situation? 

Mr. Wood. Assuming it is identified as a national interest line, 
that is the trigger. And say there were 50 lines in the country that 
the DOE puts forward, and all but one of them gets built. If the 
one does not get built, it can be built also — and this is a provision 
that was, I think, put in during the conference — it could also be 
built by someone other than the utility in the area. So you could 
have what we call a merchant transmission company come in and 
have the ability to get Federal eminent domain to build as well. 

So I do not anticipate that there will just be an absence of any- 
body willing to build the line, particularly in light of the fact that 
it has a predictable — it may be a relatively low cash flow compared 
to what generation used to be. I think I am thinking about a com- 
ment that you all made. But it is pretty steady; I mean, 12 percent, 
13 percent return on equity, predictable over time, is a lot better 
than 20 percent 1 year and 5 percent the next year that we have 
seen on the generation side. 

Senator Voinovich. So you think it 

Mr. Wood. It would be an attractive investment, I think. It is 
steady. I hate to use the term widows and orphans, but it is kind 
of what traditional utility stocks used to be. This is still a regulated 
entity. Transmission is actually highly regulated. So it would con- 
tinue forth in that regard. 

Senator Voinovich. OK; and the PUCO, the respective State, 
would be the one that would have the jurisdiction over increasing 
the rate in order to pay for the transmission line. 

Mr. Wood. Well, as it stands now, in the RTO format, which the 
Ohio utilities are part of, the rate is actually approved as part of 
a Federal rate, which the utility can then seek to recover, say it 
is Cinergy, can seek to recover that in its Ohio-regulated rate. So 
it is kind of two levels. We set a wholesale rate, transmission rate. 
We say the transmission rate is X. Then, the company pays for that 
to all of the other companies that sell transmission. And then, its 
payment is one cost like income taxes or employee or labor costs 
or new power plant costs that go into the retail rates. 
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So it is one that the Ohio commission would have ultimate say 
on what the total rate is, but the FERC component of that rate is 
a valid and effective rate in the first place. 

Senator Voinovich. So you kind of set the parameters of the rate 
on the national level, and then, it is up to the 

Mr. Wood. State how they might want to 

Senator Voinovich [continuing]. Organization to work with the 
utility as they traditionally do in terms of their rate-setting. 

Mr. Wood. Yes, sir. They might allocate the cost, for example, 
to large customers and small customers differently than the next 
State would do. 

Senator Voinovich. Senator Carper has joined us, and Senator, 
I have had a little opportunity here to ask some questions, and 
Senator Lautenberg was with us for a couple of minutes. And we 
welcome you. 

Senator Carper. Thank you. 

Senator Voinovich. And would you like to make some kind of 
opening statement before we continue the questions? 

Senator Carper. No; I am sorry I missed your questions, though. 
I would like to ask just a couple of questions. 

Senator Voinovich. Yes, sure; we welcome you. 

Senator Carper. Thanks very much. And to our witnesses, wel- 
come to this morning’s hearing. 

As we are gathered here in this hearing room, a debate is going 
on on the Senate floor, as you may know, on the energy bill that 
has been reported out of conference. Regrettably, during the con- 
ference, the Democrats in the Senate were not invited to partici- 
pate. And I believe nor were Democrats in the House, and that is 
regrettable. 

And while there are some good things in that energy bill, there 
are a number of aspects of it that are troubling to a number of us. 
Today’s hearing is, as I understand it, the second of two hearings 
that are designed to deal with the Federal role in preventing power 
outages, the likes of which we witnessed in the Northeast and Mid- 
west a couple of months ago. 

The energy bill that is before us, specifically the conference re- 
port that is before us, lays out a Federal role with respect to trans- 
mission of energy, whether you happen to be in our part of the 
country where we participate in what is regarded, I think, as a 
very successful grid network. My question of each of you is — and 
I have not read this report — but I would just like to know when 
you marry up what your beliefs are what the Federal role should 
be, and you compare that to what is proposed in the energy con- 
ference report, how close are we, how near are we to the mark in 
terms of where you think we ought to be? 

Mr. Glotfelty. First of all, the report outlines the causes. It 
does not have recommendations on how we make our system more 
reliable. This was just phase one of our investigation. Phase two, 
which will be done, and we will have a report out early in January, 
we hope, January or February of next year, will list recommenda- 
tions that we think are critical for ensuring the reliability of the 
system. 

Most of those, in my mind, will be pretty technical. They will not 
be broad policy decisions that need to be addressed by Congress. 
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We think that the majority of the broad policy issues actually have 
been addressed in the energy legislation before the Senate. First 
and foremost, mandatory reliability standards are critical. I think 
all of the witnesses here as well as the majority of the Congress 
supports mandatory reliability rules and getting them in place as 
quick as possible. 

But we also believe that other parts of the electricity title in the 
energy bill and in the tax title provide a basis for expanding our 
transmission system and making it more reliable as well; specifi- 
cally, siting provisions; tax title provisions, which encourage trans- 
mission investment; incentive base rates for transmission invest- 
ment; all are critical for ensuring that our system maintains reli- 
ability and is robust to serve our citizens. 

So we believe that it is accurate, and it is a very strong founda- 
tion for a reliable system going forward. 

Senator Carper. Thank you. Other thoughts? 

Mr. Wood. Yes, sir, Senator Carper; since I have been at the 
Commission, I have been involved in the large debate we have had 
on electricity policy, and I do think certainly, some certainty is 
called for. We really have been in kind of a stasis now for some 6 
years; certainly, the last 8 months. And we cannot really afford to 
keep going. 

So that is kind of a timing issue. As far as the substance, I do 
think the substance here is good. I think the NERC language, the 
reliability language, where you would — and this has been, again, 
kicked around for 6 years — it is time to get it down so we can get 
rules in place by next summer so that there is true accountability, 
true enforceability, a much more formal as opposed to informal or- 
ganization of the reliability enforcement across the country, be- 
cause it is a multiowner grid that really works as a single machine 
for the two halves of the country. 

To the extent that new transmission investment is required, and 
this report shows that it is not just a hardware issue; it is a human 
resources issue, too, but to the extent that investment is required, 
the incentives that are provided in this bill, which I think are actu- 
ally progressive; yes, it is not just throwing money at a problem. 
It has a very strong focus on new technologies that are involved. 
That was introduced at conference. That was not in the original 
bill. But the technology angle is one that our Commission has been 
increasingly adamant on, and to have that kind of support to 
incentivize and attract the new technologies to bring them out of 
the lab and into the marketplace and onto the poles and the wires 
and the systems that make our grid reliable are really, actually, 
progressive standards. 

The siting authority that we have just discussed with the Chair- 
man, clearly, I would rather it never be used; that the States actu- 
ally handle these siting issues, because I do think that with land 
owners in particular and environmental issues, the decisions on 
those should be as close to the people as possible. But to the extent 
that there are obstacles, either legal or bureaucratic, to getting the 
needed infrastructure in place, this bill makes it clear that that is 
going to happen, and those things should be handled on a broader 
scale than perhaps is being done. 
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I also think that the bill’s strong endorsement for regional grid 
operators, the RTOs, while not mandatory is a sufficient and quite 
important provision that this Congress goes on the record sup- 
porting competitive wholesale power markets and supports regional 
transmission organizations and that utilities should join them; it is 
very important to our Commission. I think there has been a big 
skirmish over the Commission’s desire to make those mandatory, 
which, of course, I do support but is being put on the side burner 
for 3 more years. 

Well, if in 3 years, we do not have everybody in RTOs anyway, 
then, shame on us, because this blackout report, our experience 
with economic efficiencies and Senator Lautenberg’s market that 
he talked about in his comments, which your State also is a part 
of, Senator Carper, is very compelling, and that story is one that 
the rest of the Nation is starting to understand as we kind of get 
past and learn from the California experience about how markets 
can also be done very wrong. 

And finally, when markets are done wrong, this bill provides en- 
forcement and penalties that our Commission has never had before, 
not only the ability to order refunds for cases the day that they 
happen but to actually put punitive administrative penalties on top 
of people who violate the power market rules and violate the law. 
We have less authority today than the Delaware PUC has. But yet 
we are intended to be the national regulator. So this bill corrects 
that as well. 

So there is a lot in it. You have other issues on it, but I do think 
just from the point of view of what you asked me as a regulator 
for the electricity industry, there is a compelling case to be made 
for this legislation. 

Senator Carper. Thank you. Mr. Gent. 

Mr. Gent. Senator Carper, if I could just add, NERC’s only inter- 
est in the energy bill is the reliability provisions. We have had con- 
sensus on that for nearly 4 years, and I believe that if we had this 
in place a couple of years ago, we would not have had the blackout. 
So I would urge you to do what you can to help us out by passing 
that legislation. 

Senator Carper. OK; in Delaware, we are part of one of those 
SROs that is called PJM, which we think is a model in some re- 
spects for our country. And I guess I am just especially interested 
in how the provisions of the energy conference report might affect 
the dependability of the grid, the electric grid, within PJM. We 
think we have a good system. We like the way it operates, and 
when we are losing power in a lot of other parts of the country, it 
is sort of like washed up against our region and pretty much 
stopped there, and we do not want to mess up a good thing. 

Mr. Wood. Nothing in the bill would impact — and I care a lot 
about that, too, because it is, from a national perspective, the expe- 
rience in PJM and now, more recently, in New England, which has 
adopted a very much close to PJM, the same type of market model 
and organizational model, just since I have been on the FERC; New 
York is in the process of probably by March of adopting that same. 
So you are going to have really the whole Northeast in largely the 
same format. They are close today; they will be even closer after 
New York. 
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We are very interested in that model not only surviving but 
thriving, and nothing in this bill, in my read or in anybody else’s 
read, would set that back at all. In fact, with the reliability overlay 
here, I think it enhances it, because it gives not only the economic 
oomph that we have under current law to back up economic prac- 
tices and economic decisions, but it now adds that important sister 
consideration of reliability and says they are both important; they 
are both enforceable under the national law, and that is the way 
it is going to be. 

So I think that that buttresses, actually, the capabilities of PJM 
and the other independent operators in the region. 

Senator Carper. All right. Anybody else have a thought on this? 

Mr. Gent. For at least 30 years, I have been preaching that we 
need to have, first, larger power pools, and then, the term was re- 
gional transmission organizations, and then, it was ISOs, and now, 
it is RTOs. Speaking only from an operating standpoint, I think 
that North America would be far better off if we had a dozen or 
less of these types of organizations, and certainly, PJM is the all- 
star model that we would point to in terms of operation. 

For my constituents, I have to make it clear I am not proposing 
the economic operation of PJM, but from a reliability operational 
standpoint, I think it is stellar. 

Senator Carper. Good. Thank you. Mr. Chairman, could I ask 
one more? 

Senator Voinovich. Go ahead. 

Senator Carper. This is a fairly broad question. You have been 
here testifying. Our Chairman has had the benefit of listening to 
your testimony. The reason why there are not more of us here is 
because all of us have three or four hearings going at the same 
time, and we are just trying to spread ourselves around. Some of 
us may have a press conference around 12 o’clock that we are look- 
ing forward to. 

But as I walk out of here, another point or two that you would 
want me to take along, just say if you remember nothing else or 
keep nothing else from this hearing, what would that be? 

Mr. Glotfelty. From my standpoint, I think it would be that 
this blackout is a reminder that the States and the Federal Gov- 
ernment must work together; the economic cost of a blackout of 
this magnitude is huge. We have smaller blackouts or smaller lines 
that trip every day across our system, and a renewed focus and re- 
newed attention at the Federal level within the Congress and with- 
in the States to ensuring that our system is reliable is actually crit- 
ical for our economic growth moving forward. 

We do, as Mr. Wood said, have tremendous technologies that 
have been in our labs. Entrepreneurs all across the country are 
coming to us every day with new technologies that make our sys- 
tem more robust and more reliable. And giving them the oppor- 
tunity to put those on the system, to make it a more reliable sys- 
tem, is critical moving forward. And the energy bill provides that 
in our mind. 

Senator Carper. All right, thank you. Mr. Gent. 

Mr. Gent. I believe that the legislation, particularly the reli- 
ability part of the legislation, provides for a way for the stake- 
holders to remain engaged and keep their expertise out in front 
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and fresh and involved in the standards-making and enforcement 
process with a Federal backup when needed. So I would urge you 
to do something about passing the reliability legislation. 

Senator Carper. All right. Mr. Wood. 

Mr. WOOD. I am going to echo my colleagues here on the reli- 
ability issue. As a natural gas regulator, too, I do want to point out 
how critical it is, and this bill does address it, that the Alaska nat- 
ural gas pipeline project come to be in the next decade. The avail- 
ability of reliable and environmentally-benign and domestically- 
produced natural gas is very critical to the overall economic health 
of our country. We have just seen last year, those prices have actu- 
ally doubled as supply has come, now, to more of a crunch than we 
ever thought; that has had a lot of impact on a lot of industries. 
I know some in your home State and some in mine of Texas as well 
that are very gas-intensive industries that, if we are looking into 
the future, $5, $7, or $8 gas when we have our own gas right here 
in Alaska to bring down and keep the price in the $3, $4, or $5 
range, that is a step that has to be taken, and I think it will not 
be taken unless Congress provides the kind of regulatory, legal, 
and in the case of the loan guarantees, some financial security for 
what is probably the biggest engineering project in our lifetime. 

The additional increment of liquefied natural gas to that overall 
mix is very important. These things all, this is where the electricity 
of the future is coming from. It is coal and natural gas. Yes, there 
will be nuclear; yes, there will be renewable; yes, there will be 
more hydro, perhaps, but coal and natural gas are going to be the 
two big pistons of that engine, and there are provisions in this bill. 
I know they are not beloved and all, but we have got to step back 
and think what else do we have? We are not going to put solar pan- 
els in space and beam it down like something out of a Star Trek 
movie. It is going to be coal and natural gas. So we have got to 
make sure we have got clean coal, and we have got to make sure 
we have abundant natural gas. 

And so, the steps that are laid in this bill to make that happen 
in the nonelectricity pars of this bill are real important. And I hope 
that is weighed into consideration by Members of the Senate. 

Senator Carper. When I was first reading the press reports on 
the conference report of the energy bill, among the provisions that 
raised my spirits and my hopes were the provisions dealing with 
the construction of a natural gas pipeline from Alaska. I have since 
learned that the chairman and CEO of Conoco Phillips, which is 
the oil and gas company that was believed to be most likely to par- 
ticipate in building a natural gas pipeline to bring natural gas 
down from Alaska had written to the conferees several weeks ago 
and indicated what needed to be in the bill in order for them to 
go forward with the project. 

And what his company needed to be in the bill was not included, 
and he has indicated, as I understand it, that they are not going 
to go forward with the project. 

There were several labor unions, some building and construction 
trade unions; I think the IBEW was among them; the Teamsters 
was another labor group that was strongly in support of actually 
opening up ANWR but also very much in support of the natural 
gas pipeline proposal. And I learned yesterday that they have with- 
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drawn their support from the bill because it falls short of really 
making good the commitment to build a natural gas pipeline. 

We are still trying to run this one down fully and understand it, 
but Senator Voinovich and I talked a whole lot about the need for 
natural gas and our concerns about rising natural gas prices. I was 
born in West Virginia, and my dad used to be a coal miner for a 
period of time. I have a whole lot of concern about coal there and 
other places in the country. We are the Saudi Arabia of coal here 
in America. And I want to make sure we have access to coal and 
clean coal technology to use it, and I sure want to make certain 
that we can bring that natural gas down from Alaska, and I am 
just troubled by the prospect that maybe we are not. 

Senator Voinovich. Well, I am glad you brought up some of 
these other issues. 

First of all, I think that we should all feel very good that finally, 
we are doing something about reliability and mandatory standards. 
I think in the testimony that you have given that we have had 
these things happen before, and we just ignore them until the next 
time, and I think that you should be congratulated, and I think my 
colleagues in Congress should be congratulated, that we have de- 
cided to take this on and do something about it. 

My concern is that if this bill is not passed, what are we going 

to do in the interim period of time? I mean, is there anybody 

Mr. Wood. I have made a career out of looking at statutory lan- 
guage pretty closely, and I think it would be a challenge, but cer- 
tainly, this Commission is compelled, in light of what we hear here 
in this report and what we have learned from participating on the 
task force that we would push hard to find it in the Federal Power 
Act somewhere. It is going to be a challenge, and it is going to be 
hard, but we are going to do our best to go forward under whatever 
statute we can find. And we are scrambling hard to do that, but 
I can tell you it is going to be infinitely harder to do it that way 
than if Congress says we want it this way — do it. But we will com- 
mit to trying our best under the Federal Power Act and look in the 
penumbra of the statute and find it where we can. 

Senator Voinovich. Well, it would be very important to this Sen- 
ator and to Senator Carper if you could communicate that to sev- 
eral of the Senators on my side of the aisle and perhaps some on 
Senator Carper’s side of the aisle about how important this is in 
terms of they may have some problems with other parts of this bill, 
but if you do not have this authority, you are not going to be able 
to move forward and deal with this issue that could substantially 
impact on the wellbeing of their respective States. 

Mr. Wood. I appreciate the opportunity you all have given us 

today to do that. I know the timing is pretty 

Senator Voinovich. Well, somebody ought to pick up the phone. 

We all know who they are [Laughter.] 

And try to influence them to say this is important stuff, for this 
country. The problem here in the Senate is that there is never a 
perfect piece of legislation, and too often, we let the perfect get in 
the way of something that is good and moves us down the field, 
and if it is not as good as we would like it to be, we have another 
shot at it when Congress comes back. 
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But in this case, it is not going to be done; then, if it is not going 
to be done, then, you have got to decide you have got to try to fig- 
ure out some other way you can get it done. And then, the next 
thing will be that we pull this out of this bill and then try to get 
it done next year, and you know how difficult that is going to be. 

There was a statement that if — did you make it, Mr. Gent? — the 
mandatory reliability standards had been in place, and there were 
penalties — one of you said this — that you believe this would not 
have happened. I want you to comment on it. 

Mr. Gent. Yes; I believe that we have the right standards, but 
what we do not have now are the rights to do audits, to enforce 
compliance with the standards. We have been trying for several 
years on a voluntary basis to get people to agree through contracts 
to subject themselves to reliability standards. We have been suc- 
cessful in the West of getting three standards agreed to by a wide 
group of people, and even there, certain companies have refused to 
sign up and allow themselves to be subjected to mandatory stand- 
ards. 

So we are convinced that the only way we can really do this is 
to have a law that says you have to do it. 

Senator Voinovich. And this does get it done for you, with pen- 
alties. 

Mr. Gent. My general counsel here testified at a recent hearing 
2 years ago that it was not a question of if; it was just a question 
of when, and I think that we can state that again if we do not get 
the legislation. 

Senator Voinovich. So again, you really feel that if what is in 
this legislation was in place, in your opinion, this probably would 
not have happened. 

Mr. Gent. The reliability legislation, yes. 

Senator Voinovich. All right. 

Mr. Gent. That is my opinion. 

Senator Voinovich. The other thing that I would like — we have 
identified the responsibilities of the various parties in terms of, in 
your opinion, the cause of this. The issue is did anything, did the 
grid contribute to this, were the transmission lines adequate? It is 
like when we had the stock market crash of 1929, and it went 
down. And we know that there were certain things that happened. 
But there was something wrong with the market that allowed it to 
collapse. And since that time, we have changed some of the things 
to try to prevent that kind of thing from happening. 

We had a crash here, did we not, a big crash? And the issue is 
is the transmission system inadequate to the extent that it contrib- 
uted to this at all? Or was it strictly a matter of certain people not 
doing certain things? 

Mr. Glotfelty. I will begin that one. I think it is the latter, at 
least on this example. First Energy and MISO had the tools that 
were available. They had the responsibility to ensure that this did 
not happen. The system has worked very well every other day since 
and every day before. 

There were smaller lines that do fail, as we have said, every day. 
But the system failed that day. And that is not an indictment of 
the whole region and the transmission lines within that region on 
any other day. That was just the process and procedures and fail- 
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ures, human, computer and mechanical that happened on that day. 
Saying that, there probably are transmission lines that can be built 
in Ohio, around Lake Erie, that can provide more stability to the 
system, and that is something that I suspect the Midwest ISO and 
FERC, as well as the Ohio Public Utility Commission, are all con- 
sidering. 

Mr. Gent. But there is another aspect to your question that I 
would like to address. As we proceed in the investigation, and we 
take a closer look at all of the things that actually did happen, we 
may actually decide that we have to redesign some of the ways that 
we set relays. I hate to get too technical here, but there is a process 
underway. We might be reaching out too far. We might be tripping 
too soon or not soon enough. And all of this has to be considered 
in our committee stakeholders process. It might call for a redesign 
of certain elements. 

Senator Voinovich. In other words, if the grid had been more ro- 
bust, would that have had anything to do with this? 

Mr. Gent. I am not sure I can say. I believe that the events, as 
they transpired, would transcend any robustness. 

Senator Voinovich. So the thing is that the utilities that were 
involved in the MISO, when the new law goes into effect, you are 
going to have the mandatory reliability standards which will put 
in some discipline into those organizations to do certain things. 
And then, you will, at the same time, look at the grid to see how 
that can be also enhanced to make it even more effective in terms 
of moving electricity and responding to the kinds of things that oc- 
curred on August 14. 

I just want to say to you that I will never forget that day, be- 
cause I was coming into Cleveland with my wife, and we were not 
sure whether the plane would land. We thought that maybe the 
control tower might not let us in. And then, when we got there, it 
took us a couple of hours to get our bags, because all electricity was 
off in the place, and I will never forget it. And then, we were with- 
out electricity for 24 hours, which was not too bad, but my daugh- 
ter and many other people were without electricity for several days. 

For the every day citizen, this is a big deal. And I think that 
sometimes, we take for granted, I know after they had this hurri- 
cane here, my staff people were without electricity for 6 or 7 days. 
And it is very significant. It is a very high priority. Having reliable 
electricity is important to our quality of life and to our standard 
of living and also reflective on our economic well being. 

And one of the reasons why we have been so successful as a Na- 
tion is that we have had reliable electricity at reasonable costs, and 
it seems to me that if you look at the cobweb and the maze out 
there of all of the things that the utilities in this Nation are con- 
fronted with that we need to streamline the whole process. I think 
what we are doing here will streamline it; the fact that there are 
going to be incentives to build transmission lines; that it is going 
to be a little bit easier to site them and move forward with them 
is good. 

But other issues, Mr. Wood, are very important: Things like new 
source review that has been kicking around, and finally, the ad- 
ministration has had the courage to take it on and is being criti- 
cized, vastly by many of the environmental groups. But utilities 
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were in limbo. They did not know what to do, whether to move for- 
ward or not. So they did nothing. It did not make their operations 
more efficient and did not do anything more to improve in terms 
of the quality of the environment. 

And then, we have the whole issue that has been kicking around 
here for the last several years in terms of the 4-P bill and how we 
deal with NO x , SO x , mercury and then deal with the issue of green- 
house gases. And I do not think people appreciate the fact that all 
of these things that are going on make it very difficult for us to 
get through. It is almost like the Maginot Line, trying to figure out 
how you can get anything done. 

And I would urge all of you in your respective organizations to 
take a little more interest — I am on the Environment and Public 
Works Committee, and this hearing could have been held there. 
But the fact is that we need to start to have a much more global 
look at — U.S. look at how all of these things connect up with each 
other and bring some sense and some certainty to a very uncertain 
environment that we have had for too long. And I think it is really 
important that we get our environmental groups, harmonize our 
energy, and our environmental concerns in this country and that 
we start talking to each other instead of talking past each other. 

This is very serious business, and I can tell you right now it is 
impacting on my economy in my State. You talked about natural 
gas. We are losing business after business from our State, because 
they are going — some of them are going overseas because of their 
natural gas costs are being lowered. 

We had a situation where we have done everything we can to 
close down the availability of natural gas and exacerbate the de- 
mand for natural gas. And the prices have skyrocketed. The people 
have got to understand that that impacts on not only our busi- 
nesses, our manufacturers, on agribusiness, on the chemical indus- 
try; it also impacts on just ordinary citizens: People, particularly, 
who are what I refer to as the least of our brethren, the elderly 
and those people, who are poor. 

All of this has got to be taken into consideration as we deal with 
this. And so, I would say to you you have your respective respon- 
sibilities, but I think it is also incumbent on you, the Department 
of Energy, Mr. Gent at NERC, the FERC to start to connect up 
with some of the other agencies to start looking at the big picture 
and maybe come back with us and say look, this is what we are 
going to need if we are going to have an environment where we can 
provide reliable electricity at reasonable cost and, at the same 
time, make sure that we protect our environment. 

So I want to thank you very much for being here today. We will 
be back again after the report is finished, and I am interested in 
that final report, and I think it is important that you give everyone 
an opportunity that has some issues with it to be heard so that 
their points of view are recorded there, or they feel like they have 
had their “day in court.” And the next time we get together, I 
would also, hopefully, this legislation will have passed, and if it has 
not, then, we will have to decide on how we are going to get this 
job done together. Thank you very much. 

[Whereupon, at 11:28 a.m., the Subcommittee adjourned.] 
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Good morning, and thank you for inviting me to address the status of the U.S.- 
Canada Power Outage Task Force investigation into the August 14 th blackout, as well as 
the Administration’s views on how we can ensure a more robust transmission system in 
the future. 


It’s been just under a month since the widespread power outage that temporarily 
disrupted life and economic activity in large segments of the northeastern and mid- 
western United States, and parts of the Canadian province of Ontario. And in that month, 
we have made good progress in our effort to determine the causes. 

Within a few hours of last month’s blackout. President Bush and Prime Minister 
Chretien ordered a cooperative investigation into the incident. Top government officials 
from both countries - and scores of technical and engineering experts - have been hard at 
work ever since to determine exactly what caused this outage, how it was allowed to 
spread to such a large area, and what can be done to reduce the chances of such an 
incident in the future. 
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The Task Force is co-chaired by Energy Secretary Abraham and Canadian 
Natural Resources Minister Herb Dhaliwal. The U.S. members of the Task Force are 
Tom Ridge, Secretary of Homeland Security; Pat Wood, Chairman of the Federal Energy 
Regulatory Commission; and Nils Diaz, Chairman of the Nuclear Regulatory 
Commission. The Canadian members are Deputy Prime Minister John Manley; Kenneth 
Vollman, Chairman of the National Energy Board; and Linda J. Keen, President and CEO 
of the Canadian Nuclear Safety Commission. 

The Task Force has been working on a number of fronts to collect and assess 
information on the blackout, visiting power plants and control facilities, interviewing grid 
operators and utility personnel, and analyzing vast amounts of computer and 
communications data relating to the incident. 

The investigation team is making good progress with the formulation of a timeline 
of events that led up to the blackout. That detailed sequence of occurrences will serve as 
the primary framework for piecing together all the facts and events that will lead us to 
definitive answers about what happened. 

The Task Force is gathering and analyzing information on tens of thousands of 
individual events that happened across thousands of square miles. All that information is 
being collected, compiled, sequenced and verified so we can be sure that our conclusions 
are complete, correct and credible. 
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It’s an extremely complex undertaking to analyze and understand all these 
simultaneous events on such a large expanse of the grid. This outage took about 34,000 
miles of our nation’s 1 50,000 miles of high-voltage transmission lines out of service. 
More than 290 power generation units were tripped off line or shut down. Thousands of 
substations, switching facilities, circuit-protection devices, and other pieces of specialized 
equipment were affected, and a very large number of people, policies and procedures 
were involved. 

To expedite the complicated work of sorting through all this, the U.S.-Canada 
Task Force is organized into three Working Groups focusing on specific aspects of the 
August 14 th outage. 

Our Electric System Working Group, led by experts at the Energy Department 
and the Federal Energy Regulatory Commission along with Natural Resources Canada, is 
focusing on the transmission infrastructure and its workings and management. 

The Nuclear Power Working Group, managed by the Nuclear Regulatory 
Commission and the Canadian Nuclear Safety Commission, is looking at how nuclear 
plants in the affected area performed during the outage. 

And our Security Working Group, managed by the Department of Homeland 
Security and the Canadian government’s Privy Council Office, is looking at all the 
security aspects of the incident, including cyber security. 
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Each Working Group also consists of technical, management and engineering 
experts appointed by the governors of each U.S. state affected and the Province of 
Ontario, in addition to the governmental agencies involved in the investigation. 

In addition to the Department of Homeland Security, the Security Working Group 
also includes agents of the U.S. Secret Service and the F.B.I, as well as experts from the 
Department of Energy laboratories. From Canada, the Security Working Group includes 
representatives of the Office of Critical Infrastructure Protection and Emergency 
Preparedness, the Royal Canadian Mounted Police, the Canadian Security Intelligence 
Service and the Ontario Ministry of Public Safety and Security. 

The Nuclear Power Working Group is visiting the U.S. and Canadian nuclear 
power facilities that were affected by the outage and examining their performance during 
the incident. So far, the Working Group has been able to determine that all the nuclear 
plants shut down automatically when power disturbances were detected on the grid - 
performing exactly as they were designed. 

It is a testament to the scale of the event on August 14 lh that of the 103 nuclear 
power plants operating across the United States, 70 plants detected some level of grid 
disturbance but accommodated the fluctuations and remained on-line. All the affected 
nuclear plants are now all back on-line and performing normally. 
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The Electric System Working Group has the largest challenge in the investigation 
because of the sheer size and complexity of the infrastructure. This group, working with 
technical experts at organizations such as the Independent System Operators from the 
affected regions and the North American Electric Reliability Council - an industry 
association formed following a major 1965 blackout to help assure grid reliability — will 
be looking at the flow of events surrounding the blackout and determining how they are 
interrelated. This team also is focusing on the control mechanisms that were designed to 
keep the blackout from spreading to other areas. 

Technical support for the Electric System Working Group is being provided by 
the Energy Department’s Consortium for Electric Reliability Technology Solutions - a 
group of experts from our national laboratories, and a number of universities, with broad 
experience in transmission and power delivery issues. 

This team, which has investigated a number of major power outages including the 
1999 blackouts in the West, includes some of the world’s foremost experts in 
transmission reliability issues, grid configuration, transmission engineering, wholesale 
power markets, outage recovery and power system dynamics. In addition, we have 
recruited transmission experts from the Bonneville Power Administration to help in the 
investigation. These experts led the team that examined the 1996 blackouts in the West. 

This group also was instrumental in producing last year’s comprehensive National 
Transmission Grid Study that outlined the requirements for bringing our transmission 



66 


6 

system up to 21 s1 century standards. These technical and engineering professionals are 
devoting their full attention to the work of the Task Force to help ensure an efficient and 
high-quality investigation. 

In addition to putting together the timeline and analyzing data from control 
centers and other sources, our investigators have completed on-site interviews with most 
of the control room operators of the affected utilities. We hope to complete the first 
round of interviews very soon. 

Once we have determined the causes of the blackout, we will enter Phase 2 of the 
Task Force’s two-part assignment, which is formulating recommendations to address the 
specific problems we uncover. Any recommendations the joint U.S. -Canada Task Force 
makes will likely focus on technical standards for operation and maintenance of the grid, 
and on the management of performance of the grid, in order to more quickly correct the 
problems we identify. 

We are determined to complete this inquiry in a timely manner. We hope to have 
conclusions and recommendations in a matter of weeks - not months. As Secretary 
Abraham has said, we will not compromise quality for speed. We want answers quickly, 
but we want to make sure they are the right answers. 

Beyond the investigation of the specific causes of the August blackout and the 
Task Force’s recommended remedies for those causes, there is also the broader focus on 
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the federal role in electricity reliability. Both the Department of Energy and the Federal 
Energy Regulatory Commission have launched a number of initiatives aimed at making 
our nation’s transmission grid more efficient and reliable. 

The President’s National Energy Policy noted that one of the greatest energy 
challenges facing America is the need to use 2 1 5 ' century technology to improve our 
nation’s aging energy infrastructure, which has not kept pace with growing demand or 
with fundamental changes in our electricity markets. 

Since the President’s first days in office, the Administration has strongly 
supported proposals to establish mandatory and enforceable reliability rules to reduce the 
risk of power outages. Wc are pleased that both the House and Senate included 
provisions to establish mandatory and enforceable reliability standards. 

The Administration also has supported proposals that would expand investment in 
transmission and generation facilities by repealing the Public Utility Holding Company 
Act, which has limited the resources that could be invested in the transmission system by 
restricting certain types of investors. The Administration supports provisions to advance 
the development and deployment of new technology necessary to fully modernize the 
grid, such as higher-capacity power lines and advanced monitoring and communications 


equipment. 



The National Energy Policy also called for the transmission Grid Study that was 
completed in May 2002. The study outlines the current condition of our transmission 
system and recommends ways to promote the expansion of overall transmission capacity, 
elimination of transmission bottlenecks, enhancement of the grid’s technical efficiency 
and improvement of the system’s overall reliability. 

The Grid Study’s recommendations included establishing an Office of Electric 
Transmission and Distribution at the Department of Energy, which is now helping lead 
the investigation of the August blackout. The recommendations also include developing 
new technologies such as superconductivity, which will allow more electricity to be 
shipped over smaller wires. 

The Administration strongly supports measures to provide greater regulatory 
certainty for transmission expansion, including provisions in the House version of H.R. 6 
providing for last-resort federal siting authority for high-priority transmission lines; and 
providing for the coordination and streamlining of transmission permitting activities 
across federal lands. 

The Administration also supports options that would allow for increased rates of 
return on new transmission investments, including clarifying that FERC has the authority 
to provide incentive-based rates to promote capital investment in new transmission and 
technological upgrades to existing transmission. 
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We support the goal of regional coordination and planning through the 
mechanism of voluntary regional transmission organizations that would provide certainty 
to the marketplace, prevent undue discrimination, and assist in eliminating transmission 
constraints. 

The Administration supports provisions to increase civil and criminal penalties for 
violations of the Federal Power Act. And we support changes in federal tax law to allow 
the recognition of gain over eight years for the sale or disposition of transmission assets 
as part of restructuring and to allow rural electric cooperatives to provide open access to 
their transmission systems without losing their tax-exempt status. 

Investment in our electric transmission system has lagged behind the needs of the 
marketplace. Action is needed now to help the investment in the grid catch up to the 
growth in electricity demand and the new requirements of the competitive wholesale 
power markets, which are saving consumers billions of dollars each year. Private 
industry and federal, state and local governments must work together to ensure that our 
electricity transmission system will meet the nation’s needs for reliable and affordable 
electricity in the 21 st century. 

In addition, government research and development has a role to play. That is why 
the President’s ’04 budget request includes additional funding for high-capacity 
transmission line technologies such as superconductivity and for real-time grid- 
management tools and other transmission-enhancing initiatives. 
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Our electric delivery system is the backbone of the U.S. economy. While 
investing in the necessary upgrades seems expensive, the cost is just a small fraction of 
the overall economy that it supports. We cannot afford to let such a vital component of 
our infrastructure fail to meet the nation’s growing needs. Thank you for inviting me here 
today, and I would be pleased to answer any questions. 

# # # # 
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the Federal Workforce, and the District of Columbia, 
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United States Senate 
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The United States-Canada Joint Task Force, with assistance from the 
Federal Energy Regulatory Commission (FERC or the Commission) and others, is 
working to identify the cause of the blackout and the steps needed to prevent 
similar events in the future. Analysis of the blackout is ongoing; the cause of the 
blackout and the reasons for its broad cascade through eight states and parts of 
Canada remain the initial goal of the Task Force’s efforts. 

The federal role in electricity reliability is the focus of this hearing. In the 
electric power industry, FERC acts primarily as an economic regulator of 
wholesale power markets and the interstate transmission grid. In this regard, 

FERC is acting to promote a more reliable electricity system by: ( 1 ) promoting 
regional coordination and planning of the interstate grid through regional 
independent system operators (ISOs) and regional transmission organizations 
(RTOs); (2) adopting transmission pricing policies that provide price signals for 
the most reliable and efficient operation and expansion of the grid, and (3) 
providing pricing incentives at the wholesale level for investment in grid 
improvements and assuring recoveiy of costs in wholesale transmission rates. 

The Commission’s efforts to strengthen the interstate transmission grid 
could be further buttressed in the energy bill, now in conference. There are 
several provisions in the two electricity titles that would do so: a system of 
mandatory reliability rules established and enforced by a reliability' organization 
subject to Commission oversight; Congressional support for the formation of RTOs 
across the nation; greater legal certainty for the Commission's efforts to adopt rate 
incentives for transmission or other investment to alleviate congestion on the grid, 
including new transmission technologies; tax incentives for transmission owners to 
join RTOs and to construct new transmission; and, federal backstop transmission 
siting authority for certain backbone transmission lines, in the event a state or local 
entity does not have authority to act or does not act in a timely manner. 
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September 10, 2003 

I. Introduction and Summary 

Thank you for the opportunity to testify on the blackout experienced in the 
Midwest and Northeast on August 14, 2003, the current federal role in managing 
and regulating the generation and the transmission of electricity, and steps to 
ensure that we do not experience another incident of this nature. 

The August 14, 2003 power blackout serves as a stark reminder of the 
importance of electricity to our lives, our economy and our national security. All 
of us have a responsibility to do what we can to prevent a repeat of such a 
blackout. The United States-Canada Joint Task Force (Task Force), with 
assistance from the Federal Energy Regulatory Commission (FERC or the 
Commission) and others, is working to identify the cause of the blackout and the 
steps needed to prevent similar events in the future. While analysis of the blackout 
is ongoing, it is too early to be sure what caused the blackout or why the blackout 
cascaded through eight states and parts of Canada. 

Even at the start of this investigation, however, this much is clear: our 
electrical system operates regionally, without regard to political borders. Electrical 
problems that start in one state (or country) can profoundly affect people 
elsewhere. Preventing region-wide disruptions of electrical service requires 
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regional coordination and planning, as to both the system’s day-to-day operation 
and system upgrades. 

II. Steps Taken by FERC in Response to the August 14 Blackout 

FERC staff based in Washington, D C., and at the Midwest Independent 
System Operator (M ISO) in Carmel, Indiana, have monitored blackout-related 
developments from the first minutes. 

Immediately after the blackout began, FERC staff members went to the U S. 
Department of Energy' (DOE) to coordinate our monitoring with DOE's emergency 
response team. At about the same time, FERC staff in the M1SO control room 
began monitoring and communicating the events around the clock until most of the 
power was restored. 

During this time, FERC staff was involved in nearly 20 North American 
Electric Reliability Council (NERC) telephone conference calls with the reliability 
coordinators, assessing the situation. These calls also involved close coordination 
with our Canadian counterparts. Also, the on-site staff monitored other calls 
between M1SO, its control areas, transmission-owning members, and other ISOs 
and RTOs in their joint efforts to manage the grid during restoration. 

In Washington, D.C., FERC staff immediately mobilized to provide relevant 
infonnation to the Commissioners and to others, including DOE. These 
communications included, for example, data on output by generating facilities 
and markets adjacent to the blackout area. FERC also gathered information from 
ISO and RTO market monitors for each of the ISOs or RTOs in the affected 
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regions. Our staff closely tracked the markets to make sure that no one took 
advantage of the situation to manipulate the energy markets. Working with the 
market monitor for the New York Independent System Operator (NY1SO), 
we tracked the New York market especially closely during the period when that 
market was coming back on line and during the first unusually hot days later in the 
week of August 18. 

Currently, members of the Commission’s staff are assisting the Task Force 
on its investigation of the blackout. The Commission will contribute resources to 
this effort as needed to ensure a thorough and timely investigation. If any issues 
arising from the investigation merit specific Commission action, we will undertake 
such action independently in accordance with our statutoiy mandate 
III. Background 

A. The Current State of the Electricity Transmission Grid 

The Nation's transmission grid is an extremely complex machine. In its 
entirety, it includes over 150,000 miles of lines, crossing the boundaries of utilities 
and states, and connecting to regions outside our national borders. The total 
national grid delivers power from more than 850,000 megawatts of generation 
facilities. The grid is operated by utility staff at some 150 round-the-clock control 
centers. The large number of these centers - some relatively small — has been the 
focus of much attention in post-Blackout analysis and discussion. 

When a generating facility or transmission line fails, the effects are not just 
local. Instead, the problem often has widespread effects and must be addressed by 
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multiple control centers. The utility staff at these centers must quickly share 
information and coordinate their efforts to isolate or end the problem. Given the 
speed at which a problem can spread across the grid, coordinating an appropriate 
and timely response can be extremely difficult without modem technology. 

Transmission capital investments and maintenance expenditures have 
steadily declined in recent years. In the decade spanning 1988 to 1997, 
transmission investment declined by 0.8 percent annually and maintenance 
expenditures decreased by 3.3 percent annually. (Maintenance activities include 
such items as tree-trimming, substation equipment repairs, and cable replacements, 
all of which affect reliability). During this same period, demand increased 2.4 
percent annually. 

Transmission is a relatively small part of the overall electric power cost 
structure, accounting for only 7 percent of a typical end-user's bill. Generation, by 
contrast, accounts for over two-thirds of the customer’s bill. An integrated 
company, owning both generation and transmission assets, could seek recovery of 
new transmission investment in its rates. But given that transmission is such a 
small pan of the overall rate, a typical utility is unlikely to file to recover for just 
new transmission investment, particularly those expansions that may benefit 
another utility’s customers. 

Even more important than adding transmission capacity is improving the 
tools available to control center staff for operating the grid. One example is 
installing state-of-the-art digital switches, which would allow operators to monitor 
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and control electricity flows more precisely than the mechanical switches used in 
some areas. Installing additional monitoring and metering equipment can help 
operators better monitor the grid, detect problems and take quicker remedial action. 
Improved communication equipment can help control centers coordinate efforts 
more quickly. The level of investment in these technologies has been varied. 

B. Today’s Regulatory Framework 

Currently, there is no direct federal authority or responsibility for the 
reliability of the transmission grid. The Federal Power Act (FPA) contains only 
limited authorities on reliability. 

For example, under FPA section 202(c), whenever the U.S. Department of 
Energy (DOE) determines that an “emergency exists by reason of a sudden 
increase in the demand for electric energy, or a shortage of electric energy or of 
facilities for the generation or transmission of electric energy ... or other causes,” 
it has authority to order “temporary connections of facilities and such generation, 
delivery, interchange or transmission of electric energy as in its judgment will best 
meet the emergency and serve the public interest.” Secretary Abraham used this 
authority immediately after the Blackout to energize the Cross-Sound Cable 
between Long Island and Connecticut. 

Under FPA sections 205 and 206, the Commission must ensure that all 
rates, terms and conditions of jurisdictional service (including “practices” affecting 
such services) are just, reasonable and not unduly discriminatory or preferential. 
These sections generally have been construed as governing the commercial aspects 
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of service, instead of reliability aspects. However, there is no bright line between 
“commercial practices” and “reliability practices.” 

The explicit authorities granted to the Commission in the area of reliability 
are vety limited. For example, under FPA section 207, if the Commission finds, 
upon complaint by a State commission, that “any interstate service of any public 
utility is inadequate or insufficient, the Commission shall determine the proper, 
adequate or sufficient service to be furnished,” and fix the same by order, rule or 
regulation. The Commission cannot exercise this authority except upon complaint 
by a State commission. 

The Public Utility Regulatory Policies Act of 1978 (PURPA) also provides 
limited authority on reliability. Under PURPA section 209(b), DOE, in 
consultation with the Commission, may ask the reliability councils or other persons 
(including federal agencies) to examine and report on reliability issues. Under 
PURPA section 209(c), DOE, in consultation with the Commission, and after 
public comment may recommend reliability standards to the electric utility 
industry, including standards with respect to equipment, operating procedures and 
training of personnel. 

Since the electric industry began, reliability has been primarily the 
responsibility of the customer’s local utility. Most utilities have been accountable 
to state utility commissions or other local regulators for reliable service. Typically, 
the utility keeps statistics on distribution system interruptions in various 
neighborhoods, inspects the transmission system rights-of-way for unsafe nee 
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growth near power lines, and follows industry requirements for “reserve” 
generation capability to cover unexpected demand growth and unplanned outages 
of power plants. Many state and local regulators exercise the authority of eminent 
domain and have siting authority for new generation, transmission, and distr ibution 
facilities. 

In 1965, President Johnson directed FERC's predecessor, the Federal Power 
Commission (FPC), to investigate and report on the Northeast power failure. In its 
report, the FPC stated: 

When the Federal Power Act was passed in 1935, no specific provision was 
made for jurisdiction over reliability of service for bulk power supply from 
interstate grids, the focus of the Act being rather on accounting and rate 
regulation. Presumably the reason was that service reliability was regarded 
as a problem for the states. Insofar as service by distribution systems is 
concerned this is still valid, but the enormous development of interstate 
power networks in the last thirty years requires a reevaluation of the 
governmental responsibility for continuity of the service supplied by them, 
since it is impossible for a single state effectively to regulate the service 
from an interstate pool or grid. 


Northeast Power Failure. A Report to the President by the Federal Power 
Commission , p. 45 (Dec. 6, 1965). 

In response to the 1965 power failure, the industry formed the North 
American Electric Reliability Council (NERC). NERC is a voluntary membership 
organization that sets rules primarily for transmission security in the lower 48 
states, almost all of southern Canada, and the northern part of the Baja peninsula in 
Mexico. More detailed rules are prescribed by ten regional reliability councils. 
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which are affiliated with NERC. However, neither NERC nor the ten regional 
reliability councils have the ability to enforce these rules. 

IV. Current Commission Activities 

The reliability of the grid can be bolstered through regional planning and 
operation of the transmission system, such as regional planning of new facilities; 
greater investment in infrastructure; and better methods of monitoring and 
managing transmission flow in order to relieve congestion. The Commission has 
underway several initiatives to address these issues, including: ( I ) promoting the 
formation of independent regional organizations with clear wholesale market rules 
to promote an efficient, reliable wholesale marketplace; (2) authorizing incentive 
rates for new infrastructure, including innovative technologies; and (3) identifying 
problems in the transmission infrastructure. 

First, with respect to operating the interstate transmission grid, in Order No. 
2000, the Commission identified the benefits of large, independent regional entities 
to operate the grid, and strongly encouraged, but did not require, utilities to join 
together to form such entities. The Commission noted that such entities would 
improve reliability because they have a broader, more regional perspective on 
electrical operations than a stand-alone utility In addition, some 130 control area 
operators currently manage the operation of the transmission grid, whereas a 
smaller number of regional organizations could more effectively manage the grid. 
Further, unlike utilities that own both generation and transmission, RTOs are 
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independent of market participants and, therefore, lack a financial incentive to use 
the transmission grid to benefit one market participant. 

In Order No. 2000, the Commission recognized that regional organizations 
also have unique advantages to assist in regional planning for transmission 
infrastructure. The Commission required that RTOs have a regional planning 
process to identify and arrange for necessary transmission additions and upgrades. 
Second, almost half of the electric load in the country is being served by utilities 
which are part of an independent system operator or RTO. (The major distinction 
is the size of the entity: an ISO can be smaller than an RTO). 

In a July 2002 Notice of Proposed Rulemaking (the Standard Market Design 
Rule), the Commission proposed to complete the nation-wide transition to 
independent grid operators, building upon numerous public hearings on best 
practices in power markets around the world, and also upon lessons learned from 
market failures in California in 2000. In response to over 1000 filed comments to 
the rulemaking, the Commission issued a White Paper in April 2003, streamlining 
the rulemaking effort by identifying the key elements of market design platform for 
improving the efficiency of wholesale markets. Such a platform would, among 
other things: ( I ) promote investment in transmission infrastructure, including new 
technology and in institutional infrastructure such as regional organization with 
good market rules and customer protection; (2) provide greater regulatory certainty 
to make it safe to invest in new transmission infrastructure including new 
technology; (3) require reliable and efficient management of the use of 
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transmission within the region and between neighboring regions, through day- 
ahead markets, facilitation of demand response, and the use of price signals. 

Second, the Commission has proposed the use of incentive rates to 
encourage the efficient expansion of the transmission grid. For example. Order 
No. 2000 recognized that transmission incentives were appropriate for public 
utilities that joined an RTO and offered various incentives. 

In January 2003, the Commission sought to give additional guidance on 
these transmission incentives by issuing a proposed Pricing Policy for Efficient 
Operation and Expansion of the Transmission Grid. The proposed incentives 
would help encourage needed investment in transmission infrastructure and 
improve grid performance through: an incentive adder for all public utilities equal 
to an additional 50 basis points on its return on equity for transfer of operational 
control of transmission assets to an RTO; an additional 150 basis points for sale of 
transmission assets to an entity independent of any market participant: and an 
additional 100 basis points for investments in new transmission facilities. The 
Commission also sought comment on whether incentives for new transmission 
investment should be structured to encourage the use of new technologies that can 
be installed relatively quickly (he., do not require a long siting process for 
procurement of new right-of-way, accommodate modular and portable application, 
and may be environmentally benign). Such technologies appear to offer significant 
promise of expanding grid capacity, reducing congestion, improving reliability. 



82 


-ii- 

and enhancing wholesale competition without great cost or delay. The 
Commission is currently considering comments on the proposed policy statement. 

The Commission has also acted in individual cases to provide incentives for 
development of transmission infrastructure. For example, in June 2002, the 
Commission approved a proposal to construct transmission facilities to ease the 
constraints on Path 15 within California. The Commission authorized a premium 
on return on equity (13.5 percent) and accelerated depreciation for this project as 
an incentive for construction. 

Also, in Southwest Connecticut, an area experiencing significant 
transmission congestion, the Commission has authorized New England-wide 
rolled-in rate treatment of certain transmission upgrades and additions that were 
completed within a specified time period in order to provide incentives for the 
timely construction of these facilities. 

Finally, the Commission has adopted various procedures for identifying 
areas that need additional investment in transmission facilities. The Commission 
has conducted a series of regional public conferences to discuss the state of the 
energy infrastructure within each region, fe„ the West, Midwest, Northeast, and 
South. We intend to hold public conferences in these regions every year. State 
officials actively participate in these conferences. These conferences provide a 
forum for discussing the adequacy of the electric transmission infrastructure within 
the region, the level of transmission congestion, and potential benefits of 
increasing transmission infrastructure. 
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V. What Congress Can Do To Help 

Currently, the Congress has before it, in conference, energy legislation 
which could address a number of issues that have arisen in the debate in the last 
few weeks over reliability in our wholesale power markets. 

First, both the House and Senate bills going to conference provide for 
mandatory reliability rules established and enforced by a reliability organization 
subject to Commission oversight. Many observers, including NERC and most of 
the industry itself, have concluded that a system of mandatory reliability rules is 
needed to maintain the security of our Nation’s transmission system. I agree. 

That leads to the question of what entity will be in charge, on a day-to-day 
basis, of administering the mandatory reliability rules that are developed by the 
independent reliability authority. In Order No. 2000. the Commission identified 
the benefits of large, independent regional entities, or RTOs, in operating the grid. 
(See Appendix for exceipts from FERC Order No. 2000 on reliability benefits of 
RTOs). Such entities would improve reliability because they have a broader 
perspective on electrical operations than individual utilities. Further, unlike 
utilities that own both generation and transmission, RTOs are independent of 
market participants and, therefore, lack a financial incentive to use the transmission 
grid to benefit their own wholesale sales. 

In the seven years since the Commission ordered open access transmission 
in Order No. 888, the electricity industry has made some progress toward the 
establishment of RTOs, entities that combine roles relating to reliability. 
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infrastructure planning, commercial open access and maintenance of long-term 
supply/demand. The House bill endorses this effort in a “Sense of the Congress” 
provision. Congress can direct this effort to be completed. 

While coordinated regional planning and dispatch are sensible steps to take, 
we still need to attract capital to transmission investment. 1 understand that there is 
significant interest in investing in this industry already; however, to the extent the 
Commission needs to adopt rate incentives for transmission or other investment to 
alleviate congestion on the grid, including new transmission technologies, we 
should do so. While the Commission has recently taken steps in this direction, 
action by Congress providing more legal certainty on this issue, and in repealing 
the Public Utility Holding Company Act, can provide greater certainty to investors 
and thus encourage quicker, appropriate investments in grid improvements. 

In addition to ratemaking incentives from the Commission, Congress can 
also provide economic incentives for transmission development Changing the 
accelerated depreciation from 20 years to 15 years for new electric transmission 
assets is an appropriate way to provide such an incentive. Similarly, Congress can 
provide tax neutrality for utilities wishing to transfer transmission assets to RTOs. 

To the extent that lack of assured cost recovery is the impediment to grid 
improvements, regional tariffs administered by RTOs are an appropriate and well- 
understood vehicle to recover these costs. The Commission has accepted different 
regional approaches to pricing for transmission upgrades, but the important step is 
to have a well-defined pricing policy in place. 
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Getting infrastructure planned and paid for are two of the three key steps for 
transmission expansion. The third step is permitting. States have an exclusive role 
in granting eminent domain and right-of-way to utilities on non-federal lands. 
Under current law, a transmission expansion that crosses state lines generally must 
be approved by each state through which it passes. Regardless of the rate 
incentives for investment in new interstate transmission, little progress will be 
made until there is a rational and timely method for builders of necessary 
transmission lines to receive siting approvals. Providing FERC (or another 
appropriate entity) with backstop transmission siting authority for certain backbone 
transmission lines, in the event a state or local entity' does not have authority to act 
or does not act in a timely manner, may address this important concern. 

VI. Conclusion 

Both FERC and the Congress can take steps to bolster the reliability' of our 
Nation’s interstate transmission grid. Taking the steps outlined above can help 
avoid future disruptions in our electric supply Thank you. 
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APPENDIX 


Excerpts from Regional Transmission Organizations . Order No. 2000, 65 
Fed. Reg. 809 (January 6, 2000), FERC Stats. & Regs., Regulations Preambles 
July 1996-December 2000 U 31,089 (1999), order on reh'g . Order No. 2000-A, 65 
Fed. Reg. 12,088 (March 8, 2000), FERC Stats. & Regs., Regulations Preambles 
July 1996-December 2000 U 31,092 (2000), affirmed sub nom . Public Utility 
District No. I Snohomish County Washington, et a}., v. FERC, 272 F.3d 607 (D C. 
Cir. 2002). 

Order No. 2000, 65 Fed. Reg. at 862: 

Resolving loop flow issues: An RTO of sufficient regional scope would 
internalize loop flow and address loop flow problems over a larger region. 

Managing transmission congestion: A single transmission operator over a 
large area can more effectively prevent and manage transmission congestion. 


Improving Operations: A single OASIS operator over an area of sufficient 
regional scope will better allocate scarcity as regional transmission demand is 
assessed; promote simplicity and “one-stop shopping” by reserving and scheduling 
transmission use over a larger area; and lower costs by reducing the number of 
OASIS sites. 

Planning and coordinating transmission expansion: Necessary transmission 
expansion would be more efficient if planned and coordinated over a larger region. 

Order No. 2000, 65 Fed. Reg. at 863: 

For example, we understand that there have been instances where 
transmission system reliability was jeopardized due to the lack of adequate real- 
time communications between separate transmission operators in times of system 
emergencies. To the extent possible, RTO boundaries should encompass areas for 
which real-time communication is critical, and unified operation is preferred. 

Order No. 2000, 65 Fed. Reg. at 867-68: 

The fourth proposed characteristic of an RTO is that it must have exclusive 
authority for maintaining the short-term reliability of the transmission grid under 
its control. In the NOPR we identified four basic short-term reliability 
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responsibilities of an RTO: ( 1) the RTO must have exclusive authority for 
receiving, confirming and implementing all interchange schedules; (2) the RTO 
must have the right to order redispatch of any generator connected to tr ansmission 
facilities it operates if necessary for the reliable operation of these facilities; (3) 
when the RTO operates transmission facilities owned by other entities, the RTO 
must have authority to approve and disapprove all requests for scheduled outages 
of transmission facilities to ensure that the outages can be accommodated within 
established reliability standards; and (4) if the RTO operates under reliability 
standards established by another entity (e.g., a regional reliability council), the 
RTO must report to the Commission if these standards hinder its ability to provide 
reliable, non-discriminatory and efficiently priced transmission service. 
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Mr. Chairman, Members of the Committee, my name is Alan R. Schriber. I am the 
Chairman of the Public Utilities Commission of Ohio and the Ohio Power Siting Board 
and i am here today to offer a response to the Federal role and response to the 
Blackout of 2003. i appreciate the opportunity to appear before this Committee, and I 
respectfully request that the written statement submitted under my name on behalf of 
the Public Utilities Commission of Ohio be included in today’s hearing record as if fully 
read. 

The Public Utilities Commission of Ohio is charged with the duty of regulating the retail 
rates and services of electric, gas, water and telephone utilities operating within our 
jurisdiction. Specifically, with respect to electricity, we regulate the distribution of power 
but not transmission. Additionally, since Ohio has restructured the industry, we no 
longer regulate generation. We have the obligation under State law to assure the 
establishment and maintenance of such energy utility services as may be required by 
the public convenience and necessity, and to ensure that such services are provided at 
rates and conditions which are just, reasonable and nondiscriminatory for all 
consumers. 

We are all deeply immersed into the factors and events that led up to the outages that 
occurred on August 14th. I am personally honored to be able to serve on the Bi- 
National Task Force on the Power Outage, and I am certain causes will be identified as 
expeditiously as possible. Following that, recommendations as to remedial action will 
undoubtedly be forthcoming. 
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To this point, many of the events that took place in Ohio have been documented via 
timelines. However, the entire picture of what happened August 14 th will take serious 
analysis well beyond the scope of Ohio alone. The effect of the outage on the citizens 
and businesses of Ohio were documented by Governor Taft before the House 
Commerce Committee last week. In the aftermath of the outage, the Governor charged 
my Commission with the challenge of scrutinizing events as they occurred in Ohio and 
our review will complement those of the Bi-National Task Force. 

As we pursue our quest for causes and solutions to the outage, I think we will find that 
the electrical system in this nation is by no means “third world”. It is a very complex, 
interconnected system that has in fact worked very effectively. The system operated as 
it was designed to operate on that unusual day in August. Lines tripped, plants tripped, 
and systems were isolated to prevent further blackouts, just as they were designed to 
perform. If the systems had not operated as above, not only would the loss of power 
been far more extensive, but severe damage would have resulted to our infrastructure. 

While it is reassuring that the situation was "contained” to some degree, and that 
remarkable restorations were implemented, we cannot ignore the fact that weaknesses 
exist that cal! for repair. Much like the interstate highway system, traffic patterns on the 
wires have changed, congestion has increased, and wires need to be fixed. Above all, 
we learned how vulnerable we are, and how dependent we are on our electric system. 
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You will undoubtedly hear from opponents of deregulation that states such as Ohio that 
have promoted retail competition collectively contributed to the 2003 outage. I must take 
issue with this stance. The type of competition that has been promulgated at the state 
level is one of retail competition, wherein end users purchase their power from 
marketers who, in turn, buy in the wholesale market. The grid as we know it today has 
always been the vehicle over which wholesale transactions take place. It was built to 
accommodate transactions between utilities. This is nothing new. 

Nothing has really changed that principle except for the number of wholesale 
transactions that travel the wires, which is a measure of the overall increase in the 
demand for electricity. The electrons know nothing except that the quickest way to get 
somewhere is along the shortest path. Therefore, if you are a marketer in Illinois and 
buy electricity wholesale from New Jersey, you'll write a check to the generator in New 
Jersey. However, the electrons that you end up with will come from a generator close 
by, while the New Jersey generator’s electrons will stay closer to home. That is the 
difference between the contract path and the physical path. All of this is to say that retail 
deregulation, which has been adopted by less than half the states with a modicum of 
success, should not be a relevant consideration. 

The real challenge that lies ahead, and one that Congress must confront, is molding the 
electric grid into one that can accommodate the economic realities of today. The reality 
is that demand has shifted and so to have the suppliers. Parenthetically, one should 
note that, in the aggregate , generation supply is sufficient to meet demand. The 
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problem is that supply and demand for electricity are not adequately converging through 
the grid. The reason for this misalignment is a patchwork of overseers of the grid; 
regional transmission systems, private transmission systems, and systems within the 
vertical structures of utility companies are accountable to no single boss even though 
they all interconnect at some point. 

if we had many discreet, non-interconnected systems, I suspect we would have more 
blackouts than fewer, although of less duration, since there would be no interconnected 
neighbor to help out on a hot day. On the other hand, a regionally coordinated 
transmission system with a super-large geographical footprint would enhance the ability 
to work through all kinds of contingencies, some of which are simply beyond the scope 
of smaller control areas. 

Everyone should want to see our transmission resources allocated in an optimal 
manner. I am prepared to argue that its achievement is predicated on the super-regional 
transmission system alluded to above. Tu this end, FERC is the federal agency 
endowed with the authority to make it happen. Congress should support FERC’s efforts 
to enlist participation by all transmission owners into a regional grid that recognizes the 
economies of centralized management. 

I do not know how many billions of dollars it might take to upgrade the grid, but I do 
fervently believe that whatever dollars are expended are done so most economically 
when the needs of the grid as a whole are evaluated as objectively as possible. Given 
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the myopia associated with the fragmented systems of today, dollars may be thrown at 
“fixes" that often do nothing but add an asset to the utility rate base; not only are the 
needs of the region ignored, but the utility that has determined to fence itself in does 
very little at the margin to benefit its own customers. Regional approaches must be 
adopted to appreciate the needs and recognize the benefits. 

An independently administered regional transmission system, on the other hand, could 
prioritize its investments based upon marginal benefits. Dollars would flow to the points 
on the grid that would yield the most benefits, for example, the amount of regional 
congestion that is relieved, regardless of whose “backyard” it resides. Why would a 
single state permit the construction of a high tension wire within its boundaries if there 
were not a single “drop" along the way? The answer would be that it probably would if it 
understood that the congestion relieved by the line significantly increased the level of 
unobstructed power flows within the state. The problem is in the “understanding”. The 
manager of an independent, integrated, profit maximizing transmission organization 
understands the resource optimization process because it has the bigger picture. 

In addition to rational planning, the aggregated grid system is also more likely to attract 
capital. Investment dollars move to the places where the potential yields are the 
greatest given the risks. We might conjecture that the greater the number of electrons 
that flow, the greater the dollars that flow to the construction of wires that carry those 
electrons. A unified super-regional grid maximizes power flow through the grid and 
should be politically indifferent as to the points of need located within. In contrast, sub- 
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optimal investments in electric facilities are made when a single entity, without regard 
for the region around it, is more interested in closing itself off from the greater good. 
Those who provide the dollars are more likely to follow the path of investment with the 
greatest potential for risk/return optimization, which from my point of view resides with 
the regional grid. 

One great challenge to enhancing the system is the ability to site large transmission 
lines across states. Large towers with conductors capable of carrying hundreds of 
kilovolts are generally not a welcome sight in most areas, and resistance to their 
construction is something that we’li live with indefinitely. The authority to site power lines 
today lies with the states, and therein lies a source of debate for the Congress. 

Arguments break both ways with respect to federal-versus-state siting. Most states 
stand firm in their belief that power siting is strictly a state issue, and some good 
arguments are made on their behalf. First, the state decision-makers know their 
constituents. They are most familiar with the local contacts that so often weigh in on 
siting issues. Second, the speed at which certificates are granted most certainly 
exceeds that accomplished under federal jurisdiction; our experience with pipelines 
underscores the point. Finally, state officials will bear the brunt of unpopular decisions 
regardless at what level those decisions are made. 

In Ohio, the legislature has given the Power Siting Board, which I chair, broad powers. 
Affected parties are afforded hearings, and certificates are granted only after a 
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extensive range of issues are examined. These include environmental, health, 
agricultural, and others. Our siting process compels us to take into consideration the 
effects on the region, not just the state. All told, Ohio is among the most progressive 
states in getting utility facilities up and running. 

Unfortunately, other states do not move swiftly as does Ohio in siting electric or pipeline 
facilities. Furthermore, some states are dominated by Federal lands. As a result, some 
argue that federal preemption of state siting decisions is appropriate. In a series of 
meetings under the auspices of the National Governors Association, it was decided 
that, at least in the Midwest and the East, states could agree to work together to site 
interstate transmission lines. As a compromise, the Electricity Title of the Energy Bill 
under consideration provides that the FERC shall provide a “backstop” in the event of a 
recalcitrant state. This is a logical, progressive outcome. 

I have been talking to this point about the physical conditions that bind the grid for better 
or worse. However, the economics of all of this must not go unmentioned. Different 
transmission systems, as fragmented as they might be, often employ pricing strategies 
that are inconsistent with one another. When the price of moving electricity a number of 
miles across different operating areas varies according to whose area is being crossed, 
the outcome can be quite confusing for those paying the freight. Without belaboring the 
point, another strong argument that favors super-regional management of the grid is 
pricing consistency and the concomitant higher level of economic certainty conferred 
upon users of the grid. 
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This aggregation of transmission systems or control areas is the cornerstone of the 
FERC's endeavor. To be thoroughly effective, however, it must also draw lifeblood from 
Congress as Congress deliberates its Energy Bill. It is antithetical to our interests to 
delay FERC’s attempt to implement its design for a rational transmission market. 

If Congress must do any one thing immediately, it must address the issue of system 
reliability. While the states have the authority from their legislatures to set and enforce 
rules for distribution systems, the federal government must confer power upon someone 
to do the same for the transmission system. Whether it be the North American Electric 
Reliability Council (NERC) as currently proposed in the Energy Bill, or whether it be the 
FERC, the rules of the road must be mandatory. Once in place, the enforcement of the 
rules can follow the course taken by other federal agencies. 

A unique and efficient means of enforcement of some federal rules has evolved over the 
years. Ohio, as well as other states, undertakes a number of such tasks on behalf of 
federal agencies. For example, the US Department of Transportation has very specific 
rules that speak to natural gas pipeline safety. Ohio’s Public Utilities Commission 
receives funds from USDOT to inspect and enforce those rules within the state’s 
borders. Ohio also participates in the inspection protocols for the transportation of 
hazardous materials. The same process has evolved with the Federal Railroad 
Administration which has prescribed rules for rail crossings. The Ohio Commission has 
personnel evaluating and prioritizing grade crossings for the purpose of supporting 
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communities with safety devices. Given the fact that Ohio and other states already 
support federal agencies in rule enforcement, does it not make sense to consider the 
same for the transmission of electricity? 

The events of the past couple of weeks speak clearly to the need for Congress to do 
two things. First, Congress must focus on endowing some agency or organization, e.g., 
the FERC or NERC, with rule-making authority that locks-in our quest for a reliable grid. 
Second, it must enable the FERC to move forward in its initiatives to bring about a 
physically and economically rational structure and governance to the transmission 
system. 

I appreciate the opportunity to have appeared here before you today and look forward to 
clarifying anything that I have said. 
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EXECUTIVE SUMMARY OF TESTIMONY OF 
CRAIG A. GLAZER, VICE PRESIDENT, GOVERNMENT POLICY, 

P JM INTERCONNECTION 

In his testimony before the Senate Subcommittee on Oversight of Government Management, the 
Federal Workforce and the District of Columbia, Mr. Craig A. Glazer, Vice President, Government Policy 
for PJM Interconnection detailed a “road map” for Congressional action to strengthen the interconnected 
transmission grid and to meet customer needs for reliable service and stable prices. Mr. Glazer outlined 
this road map in the context of the outage events of August 14, 2003. 

Mr, Glazer points out that the events of August 14, 2003 represent as much a crisis in confidence 
in the industry as it does a failure of the electric power grid. He notes that the outage of that day, 
although requiring critical study and analysis, must not paralyze the industry from moving forward with 
critical reforms. “We must learn from the positive experiences as well as the negative ones facing this 
industry and craft rational common sense rules that follow and respect the laws of physics which govern 
this speed of light product” Mr. Glazer stated. His testimony details the value of regional coordination 
and notes that we cannot continue using outdated solutions to meet the 21 s1 century needs of customers. 
He details a road map for Congressional legislation which includes the following points: 

1. Provide FERC with the authority it needs to ensure that regional organizations can flourish to 
plan and manage the grid in a coordinated manner; 

2. Do not discourage or strip FERC’s authority to move forward in those regions of the country that 
wish to move forward with the development of competitive markets; 

3. Ensure that the laudable goal of protecting native load does not work to repeal the anti- 
discriminatory provisions of the Federal Power Act or to otherwise balkanize the grid. A clear 
statement from Congress that native load should be protected but flexibility in how that native 
load is protected would ensure this proper balance; 

4. Whether federal or state siting is preferred, encourage regional planning processes, undertaken by 
independent RTOs with state and stakeholder input, before the power of eminent domain is 
exercised to appropriate private property to build transmission. 
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5. Reliability standards should be made mandatory, with their development and enforcement 
overseen by a public body. Deference should be provided to regional solutions that improve 
reliability for the region and for neighboring systems. 

Mr. Glazer points out that much of the PJM system was spared from the effects of the August 14 
outage as a result of protective hardware, on PJM and neighboring systems, acting as it was designed to 
protect equipment and isolate the disturbance. The operation of these protective relays had the effect of 
separating much of the PJM system from the surrounding grid and thus avoiding much of the blackout’s 
impacts except in Northern New Jersey and in the Erie, PA. area. Thereafter, PJM system operators 
worked to rebalance the system and begin the process of restoration both on the PJM system and by 
providing assistance to neighboring systems in Ohio and New York. 

Although much of the protection of the PJM system occurred automatically, Mr. Glazer explains 
that PJM’s independent regional planning process has been a critical element to designing a system which 
can both support an interconnected grid but also withstand an outage of this magnitude. He details the 
work that is underway eollaboratively by PJM and the Midwest ISO to develop a Joint Operating 
Agreement and reliability plan which will provide a much higher level of coordination and 
communication among control areas in the Midwest than exists today upon the integration of the 
Commonwealth Edison system into PJM. Finally, Mr. Glazer urges Congress to encourage the 
development of such independent planning protocols and link them to any incentives it provides for the 
construction of transmission in order to ensure that transmission construction and the use of the power of 
eminent domain is undertaken wisely and judiciously. 
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TESTIMONY OF CRAIG A. GLAZER 
BEFORE 

SUBCOMMITTEE ON OVERSIGHT OF GOVERNMENT MANAGEMENT, 
THE FEDERAL WORKFORCE AND 
THE DISTRICT OF COLUMBIA COMMITTEE ON GOVERNMENTAL 

AFFAIRS 

UNITED STATES SENATE 
SEPTEMBER 10, 2003 

Mr. Chairman and Members of the Committee: 

The events of August 14, 2003 represent as much a crisis in confidence in this 
industry as it does a failure of the electric power grid. As one who has worked in this 
industry my whole professional life, I am vitally concerned that we restore the public’s 
confidence by establishing a clear road map to move this industry forward. Of course, 
time needs to be taken to ensure careful analysis and the development of solutions which 
can be tested and retested prior to full scale implementation. And although thoughtful 
reflection is needed, we simply cannot allow the events of August 14 (as significant as 
they were) to paralyze us from moving forward. 

None of us can repeal the laws of physics which ultimately control the behavior of 
this speed-of-light product. As a result, policymakers need to drive rational public 
policy, market development and infrastructure investment which free this industry from 
mountains of red tape, constant political and legal battles over individual proposals and 
never-ending regulatory proceedings over Regional Transmission Organization (“RTO”) 
formation. These solutions also need to meet the interstate and international nature of 
this speed of light product. As a result, although I will spend part of my testimony 
addressing the specific questions you raised concerning the August 14 event, I want to 
lead with what I think is the far more pressing issue: How do we address the critical 


1 



101 


crossroads we find ourselves in today? How does Congress, as our nation’s policymaker, 
moves this industry forward through clear and coherent policies and institutions? How 
do we avoid the pitfall of unclear or internally contradictory policies slowing industry 
growth and discouraging need investment? 

To answer these questions, we can look at real facts and analyze the positive as 
well as negative experiences faced by this industry. The “bottom line” is that certain 
models of deregulation and restructuring of the industry have worked and have developed 
real value for the customer. It has been proven that restructuring and deregulation can 
work to provide real benefit to customers in the form of stable prices, increased generator 
efficiency and new demand side options for consumers. Although not necessarily the 
answer to the events of August 14, market rules and procedures can work to limit the 
adverse impacts of transmission or generation outages triggering larger events. And as a 
result of our transparent and independent regional planning process, the PJM system was 
designed to withstand and did withstand, for the most part, an outage of this magnitude. 
So as we move the industry forward, we must not throw out the baby with the bathwater 
or tie the hands of the regulator to move forward based on the positive experiences that 
have occurred during this otherwise troubled time. 

Much of the mid-Atlantic region’s ability in real time to withstand the disturbance 
of August 14 was the result, not of human intervention, but of hardware working as it 
should — hardware that was designed to protect each of our systems from outside faults, 
voltage drops and other system disturbances that threaten system stability. This system 
runs from the Delaware short to Washington County, Ohio. But in the longer run, a 
transparent planning process undertaken by an independent entity such as a regional 
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transmission organization with a “big picture” look at the entire grid, can ensure that the 
appropriate hardware is in place and that reliability is maintained proactively and at 
prudent cost to the consumer. And important market tools such as ordering redispatch of 
generation between neighboring systems, something which PJM and the Midwest ISO 
have put forward as a reliability solution in the Midwest, and which PJM and the New 
York ISO are piloting between their systems, can help alleviate the adverse impacts of 
curtailments of individual transactions. Only independent entities such as RTOs can 
undertake these solutions in a manner which will not be seen by the marketplace as 
favoring one provider over another or sacrificing one entity’s “native load” at the expense 
of another’s “native load”. 

Just as Abraham Lincoln stated that “a house divided cannot stand”, neither can 
an industry continue to rely on unchanged 20* century institutions and tools to police the 
new 21 st century world surrounding this speed of light product. Today we find ourselves 
teetering somewhere in between a traditional and restructured environment. This is a 
highly unsustainable state and cannot help to either improve reliability or attract needed 
capital for investment. Let me give an example. 

The Energy Policy Act of 2003 provides for incentives for the construction of 
vitally needed new transmission. Such investment is extremely important and Congress 
should be applauded for taking this bold step. However, in the same breath, there is 
discussion of adding provisions which would limit or suspend FERC’s ability, through 
rulemakings, to create the very institutions needed to independently and in an unbiased 
manner, plan the right location for this new investment. Absent a rational planning 
process undertaken by an independent entity such as an RTO, one that balances the need 
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for generation, transmission and demand side solutions simultaneously, we risk building 
transmission in the wrong place and appropriating private property for investments that 
don’t necessarily solve (and in some cases create new problems) for the regional grid. In 
short, if we are not careful, without the proper tools in place, we run the risk of creating 
tomorrow's stranded investment and simply throwing ratepayer money at the problem. 
By contrast, regional planning processes undertaken in an unbiased public process, 
allows the marketplace to obtain the needed information to effectuate the wise choice 
between transmission, generation and demand side solutions to meet our reliability and 
economic needs. The states in the mid-Atlantic were extremely wise during PJM’s 
formation — they insisted that before any markets are started that the RTO have in place a 
regional planning protocol. They correctly noted that as we are talking of using a power, 
which only the government can grant, to appropriate private property, we ought to ensure 
that we are exercising this powerful government authority both wisely and judiciously. 
An unbiased regional planning protocol can do just that. 

For all these reasons, we recommend that Congress undertake the following steps: 

i. Provide FERC with the authority it needs to ensure that regional 
organizations can flourish to plan and manage the grid in a coordinated 
manner; 

ii. Do not discourage or strip FERC’s authority to move forward in those 
regions of the country that wish to move forward with the development of 
competitive markets; 

iii. Ensure that the laudable goal of protecting native load does not work to 
repeal the anti-discriminatory provisions of the Federal Power Act or to 
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otherwise balkanize the grid. A clear statement from Congress that native 
load should be protected but flexibility in how that native load is protected 
would ensure this proper balance; 

iv. Whether federal or state siting is preferred, encourage regional planning 
processes undertaken by independent RTOs with state and stakeholder 
input before the power of eminent domain is exercised to appropriate 
private property to build transmission. 

v. Reliability standards should be made mandatory, with their development 
and enforcement overseen by a public body. Deference should be 
provided to regional solutions that improve reliability for the region and 
for neighboring systems. 

With this overview in mind, I will proceed to address some of the questions that 
have arisen concerning the outage of August 14: 

1. What exactly were the specific factors and series of events leading up 
and contributing to the blackouts of August 14? 

2. At what time did your company first become aware that the system 
was experiencing unscheduled, unplanned or uncontrollable power 
flows or other abnormal conditions and what steps did you take to 
address the problem? Were there any indications of system instability 
prior to that time? 

3. Which systems operated as designed and which systems failed? 
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Answer 

As noted above, the location, character and proximate cause of the initial 
disruption in the transmission and supply of electricity is the subject of an ongoing 
NERC/DOE investigation and PJM defers to that investigation. As a result, PJM will 
limit its response to actions it took on its own system both prior to and during the August 
14 outage. 

As to its own system, PJM first became aware of a disturbance on the Eastern 
Interconnection at about 4:10 pm on August 14 th . Prior to that time, August 14 th could be 
characterized as a typical unexceptional summer day in the PJM control area, with a 
typical number of lines out of service, and relatively few scheduled or unscheduled 
outages. At noon on August 14 th , NERC initiated a routine time frequency correction 
across the Eastern Interconnection in accordance with NERC operating policies, because 
the time frequency had exceeded its margin for error. PJM was properly following the 
NERC standard process, but it is mentioned in this context because it accounts for a 
frequency fluctuation in PJM data at the time the correction was implemented. 

PJM became aware of significant impacts on its system from an external 
disturbance at approximately 4:10pm, At the time of the disturbance, PJM recordings of 
telemetered load and frequency revealed an initial loss of more load than generation on 
the PJM system. Subsequently system operators reduced generation output in order to 
bring the system back into balance. PJM experienced a loss of load of approximately 
4,500 MW of its total load of approximately 61,200 MW at the time of the disturbance. 
About 4,100 MW of PJM’s lost load manifested in northeastern New Jersey, while an 
additional 400 MW of load was lost in northwestern Pennsylvania near Erie. 
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The disturbances noted by PJM at approximately 4:10pm resulted in some 
individual units going off-line in PJM and in transmission lines opening. The cascading 
effect of the outage caused PJM to lose approximately seven percent of its load, but 
automatic relay devices deployed throughout PJM in accordance with our design and 
planning criteria isolated most of the PJM footprint from the power loss. Automatic relay 
devices effectively isolated most of PJM from Ohio and New York, which were subjected 
to prolonged outages. By 4:12pm., most of the tripping of generating stations and 
transmission lines within PJM had subsided. Thereafter, PJM system operators worked 
to rebalance generation and load within the PJM system by reducing system frequency to 
a normal range. In addition, PJM system operators initiated procedures for more 
conservative operation of the system, to assure that system restoration could proceed 
more effectively. The disturbance itself played out over the course of mere seconds - 
with no real-time human intervention possible - but system operators played a vital role 
in system restoration. 

In summary, the system worked as it was designed — through the automatic 
operation of relays PJM was able to isolate problems which effectively separated it from 
the outage and “kept the lights on” for the overwhelming majority of its customers. 
Through swift operator action, PJM was able to stabilize its system and also provide 
critical support to the restoration efforts in Northern New Jersey and Northwestern 
Pennsylvania, as well as the neighboring systems in the New York, and Ohio. 

4. If events similar to those that occurred on August 14, 2003 had 
happened a year ago, would the results have been the same? If similar 
events occur a year from now, do you anticipate having to place 
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equipment and processes sufficient to prevent a reoccurrence of the 
August 14 blackout? 

Answer 

Prior to the August 14 outage, PJM and its Midwest counterpart, the Midwest ISO 
had just reached agreement on an historic Joint Operating Agreement and Reliability Plan 
that, if implemented, would bring a new level of coordination and data sharing that would 
clearly have avoided some of the communication and coordination problems that arose in 
the context of the August 14 outage. The Joint Operating Agreement and Reliability Plan 
provides for an unprecedented level of coordination and data sharing among neighboring 
systems in the Midwest. The Joint Operating Agreement detailed monitoring measures 
and specific actions that each of the large RTOs would take to clear congestion or 
reliability problems on the other’s system, at key designated flowgates. It would provide 
for actions that presently do not occur systematically in the Midwest including: 

♦ day-ahead and real-time monitoring of each RTO’s system; 

♦ detailed data exchange between the two RTOs; 

♦ emergency operations protocols; 

♦ joint planning protocols; and 

♦ mandatory redispatch of each other’s generation in order to relieve 
congestion on the other’s system. 

This Agreement, coupled with the fact that there would be just two entities, both 
with planning responsibility and a large regional look as opposed to multiple control 
areas with a more limited view of neighboring systems, would provide for an increased 
level of reliability in the Midwest and would reduce the coordination and communication 
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issues that exacerbated the problems which occurred on August 14th. The Joint 
Operating Agreement and associated reliability plan were undergoing stakeholder review 
at the time of the August 14th outage. Subsequent to that time, both PJM and the 
Midwest ISO have committed to reviewing the document in light of lessons learned from 
the August 14th outage and providing appropriate enhancements. PJM looks forward to 
review and comment by the respective stakeholders and state commissions in the area. 

That being said, PJM believes that should the Joint Operating Agreement and 
Reliability plan be allowed to move forward it would provide a model that has been 
sorely lacking in this nation relative to coordination and communication between two 
large regional entities each charged with the responsibility of ensuring reliability of the 
regional transmission grid. 

5. What lessons were learned as a result of the blackouts? 

6. How can similar incidents in the future be prevented? 

Answer 

As the DOE investigation to the causes of the blackout is first beginning, it is too 
soon to detail with specificity all of the “lessons learned” from the August 14 event. That 
being said, there are some overarching lessons of August 14 which played out 
dramatically in how different entities reacted: 

We cannot continue to use 20 th century solutions to solve 2 1 st century problems — 
In the last century, reliability was ensured through a series of loosely described 
emergency support agreements among neighboring utilities. No regional planning 
process existed and each individual utility was charged with maintaining and planning for 
the reliability of its individual portion of the grid. Although regional reliability councils 
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exist to coordinated regional efforts, such entities were neither independent of the market 
participants nor empowered to require solutions and order penalties. It is clear that these 
loose agreements and institutions of the last century will not work in the future. Rather, 
we need Congress to: 

i. encourage the development of regional transmission organizations and not 
strip or suspend FERC authority to undertake necessary generic 
rulemakings; 

ii. tie any transmission investments to the use of regional planning processes 
undertaken with the states and interested stakeholders to ensure that 
whatever transmission is incented is the “right” transmission located at the 
key location needed to ensure maximum benefit to reliability and 
economics of grid operation; 

iii. encourage and require native load protection but not tolerate 
discriminatory conduct favoring one’s own market position in the name of 
protecting one’s “native load”; and 

iv. finally, Congress should make reliability standards mandatory but avoid 
codifying statutory deference to standard-setting and enforcement in some 
regions but not others. Deference should be provided to regional solutions, 
arrived at in open stakeholder processes and with state concurrence, in all 
parts of the country while any national organization review is limited to 
ensuring that solutions arrived at on less than an interconnection-wide 
basis, promote reliability in the larger region. The negotiation of the Joint 
Operating Agreement and reliability plan between PJM and the Midwest 
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ISO, which will soon be submitted for NERC review, is an example of the 
process working at its best with NERC focusing on whether the plan 
enhances reliability between regions while avoiding the commercial 
infighting among member companies. 

For grid operators themselves, it is clear that we have to ensure that our relay 
hardware is appropriately sized, maintained and programmed to protect systems in the 
event of cascading outages. RTOs need to be more vigilant in defining their role vis-a- 
vis the local transmission owner who still owns and maintains this critical equipment. 
Agreements such as the MISO/PJM Joint Operating Agreement should be a mandatory 
“baseline” of coordination between RTOs and should provide appropriate and reciprocal 
support of adjacent systems both between market areas and where market areas abut non- 
market areas. And most of all, we need to move this industry forward with flexible 
policies that are designed to meet and restore the public’s confidence in this critical 
industry so important to our nation’s secure future. 

I thank you for this opportunity to testify and look forward to your questions. 
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Good morning, Mr. Chairman and Members of the Committee. My name is James P. 
Torgerson. I am the President and Chief Executive Officer of the Midwest Independent 
Transmission System Operator, Inc. (“Midwest ISO”). I am appearing today to offer what 
insights I can concerning the circumstances surrounding the power outages and offer suggestions 
as to what might be done in the future. 

The Midwest ISO was formed in 1998. The Midwest ISO is the nation’s first voluntary 
regional transmission organization that did not originate from a legislative mandate or against the 
back drop of a tight power pool. The Midwest ISO is also the first entity found by the Federal 
Energy Regulatory Commission to be a Regional Transmission Organization. 

The Midwest ISO’s region covers portions of fifteen states and the Canadian province of 
Manitoba. Of relevance to your inquiry here, we act as a Reliability Coordinator for three sets of 
companies. As Reliability Coordinator, the Midwest ISO monitors, plans, conducts analyses 
regarding the high voltage grid and communicates with the Control Areas in our region who 
have the primary control capabilities to open and close transmission circuits and to redispatch 
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generation. We perform this coordination function for the companies that have transferred 
functional control of their transmission systems to us. We do it through contract with the East 
Central Area Reliability Council (ECAR) for two systems that are scheduled to transfer control 
to us in the future, Northern Indiana Public Service Company and First Energy’s Northern Ohio 
system (First Energy’s eastern assets are under the control of PJM). Finally, through a contract 
with MAPPCOR we perform this service to companies in the Mid-Continent Area Power Pool 
(MAPP) region that have not transferred control of their transmission systems to the Midwest 
ISO. Three of the more than 30 companies within our reliability coordinator territory suffered 
outages in the black out - Consumers Power Company, Detroit Edison Company and First 
Energy Company, 

What exactly caused the blackout will be forthcoming from the work being done by the 
International Task Force formed by President Bush and Prime Minister Chretien of Canada. As 
Secretary of Energy Abraham’s recent press release states: “It’s a complex job we are 
undertaking. ... It’s going to take some time to compile all this information, get it all 
synchronized and sequenced, and then determine exactly what happened when - and how it’s all 
interrelated.” The Midwest ISO only has a part of the data needed to reconstruct the events and 
is not in a position to characterize the proximate cause of the blackout. The Midwest ISO is 
cooperating with the International Task Force and the General Accounting Office’s 
investigations into the matter. Likewise the reason for the cascading effect of the outages is 
unknown at this time. 

The analysis that has been done to date in the Midwest seems to indicate that there were a 
number of events in the Eastern Interconnection on August 14 lh . Some are surely related to 
separations and the substantial losses of load that occurred, and others are likely unrelated. 
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During the morning and into the afternoon, Midwest ISO personnel were in contact with various 
control area operators and PJM, the neighboring reliability coordinator about the events of the 
day, which by the afternoon had included the outages of several high voltage transmission lines. 
During the morning of August 14'\ there was no indication to the Midwest ISO of significant 
problems in our territory. During the course of the hour preceding the cascading event, after the 
loss of a large generating unit in northern Ohio had already occurred, several transmission line 
outages also occurred in the Ohio area. During this period the Midwest ISO operator was in 
contact with the neighboring Reliability Coordinator at PJM as well as control operators within 
our territory. At this point in time, the issues did not seem to implicate a regional problem. 

Things began to change at 4:09. By 4:10 Eastern Daylight Time portions of the eastern 
interconnection were separating from one another and the loss of significant load was only 
seconds or minutes away. At 4:19 the Midwest ISO initiated the first NERC coordinating call of 
the day among NERC and the regional Reliability Coordinators. These calls were repeated every 
several hours thereafter and eventually to a few times per day during the restoration. During that 
first call the issues became ascertainment of system conditions and the commencement of 
restoration activities. 

During the restoration efforts, the Control Area operators performed their responsibilities 
in linking returning generation with load to be restored. The Midwest ISO, as a Reliability 
Coordinator, played its part in analyzing the transfer capability into Michigan and Ohio to safely 
deliver power into those areas. The Midwest ISO worked with each area to ensure the individual 
area restorations would not threaten even a small-scale repeat of Thursday afternoon’s events. 
The Midwest ISO was able to relay information to Michigan about power available from Illinois 
that could safely be imported to hasten the restoration of load. Finally, the Midwest ISO, in 
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combination with the IMO and others, determined when it was safe to reestablish the ties 
between Michigan and Canada. I would also like to add that as part of our normal operations, 
the Federal Energy Regulatory Commission has stationed two of its professional employees in 
our headquarters. Among the valuable assistance that they provided, on August 14 Ih these 
federal employees allowed Midwest ISO to have a single point of interaction with various federal 
entities concerned with the outage. FERC's dissemination of the real time data from our 
headquarters allowed Midwest ISO personnel to devote greater attention to system stability and 
restoration efforts. 

As only one of the companies contributing information to NERC and DOE we do not 
have a picture of events across and adjoining the footprint of affected systems. Events occurring 
across the eastern interconnection including plant outages, voltage conditions and the operation 
of protective relay schemes will have to be evaluated before cause can be distinguished from 
effect. 1 am awaiting the results of the International Task Force formed by President Bush and 
Prime Minister Jean Chretien of Canada. However, there are some preliminary observations that 
I can share with the Committee: 

• Equipment that was designed to protect transmission lines and generators during cascading 
events operated successfully to isolate equipment before there was permanent damage to the 
equipment. This shortened the time period of the restoration efforts because, had protection 
systems not operated to protect individual components as designed, the power production and 
delivery systems could have been severely hampered for many months. 

• Automatic protection systems did keep the blackout from spreading even further. 

• Considering the size of the area impacted, the restoration proceeded in an orderly manner 
with much of the load restored within 48 hours of the initial disruption. The Control Areas 
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have primary responsibility to restore their systems while maintaining a balance of resources 
and load. The ISO/RTOs assisted in the restoration effort by ensuring equipment was not 
being put at risk of furthering cascading as generators were being brought back on-line and 
as load was being restored. The coordination among the ISO/RTOs and their member 
systems worked to assure a reliable restoration. 

I believe a key reason that the Midwest ISO was in a position to help in the restoration 
efforts was because of our broader regional view of the area. Making a few presumptions, 1 
believe the Midwest ISO will be in a better position next year to lessen the likelihood of any 
recurrence. We have before FERC a tariff that if accepted and implemented will have the 
Midwest ISO running wholesale markets, much like PJM, the New York ISO and ISO New' 
England do today. That tariff will put matters like a regional security constrained unit 
commitment and real time generation dispatch in place. Each of these additions should be of 
substantial benefit. That will give the Midwest ISO more information about generation unit 
status than we have today and add an ability to direct generator actions within the footprint. This 
market will improve reliability. Indeed a strong, reliable system is the necessary underpinning of 
a successful market. The two are not opposite poles they are two halves of what is necessary for 
reliable service to customers. 

I think all the regional entities involved have an appreciation today that communication 
between reliability coordinators and other entities has to be raised to a higher level than has been 
required or practiced in the past. At a basic level, that has already happened. The use of the 
NERC coordinating call to apprise our industry counterparts of the computer virus on August 
20 th is an example of that increased communication. Mere telephone communication; however 
does not seem adequate for the future. The Midwest ISO and PJM have a Joint Operating 
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Agreement under development that calls for substantial real time automated data transfers 
between our systems. While the Joint Operating Agreement is not yet finalized, the Midwest 
ISO and PJM have recently established the physical communication network links to allow for 
the types of data transfer called for by the Agreement. Once the software is in place the 
enhanced data transfer can be made operational. We are each reassessing the Agreement to 
determine what additional features it should have in light of the events of August the 1 4 lh . 

The Committee is also confronting the question of what can be done to prevent a 
recurrence of the outages. While the definitive answer cannot be given today, I believe that you 
will find agreement that widespread adherence to strong enforceable reliability standards will be 
important. Equally important is state and federal cooperation in transmission siting. I am 
pleased that the states in the Midwest have formed an organization, the Organization of MISO 
States, to work cooperatively with the Federal government on, among other things, siting of 
transmission facilities. We believe such an approach holds great promise to allow the siting of 
needed transmission facilities while protecting regional efforts to address issues associated with 
wholesale electricity markets and reliability. 

Other matters will be crucial as well. In my opinion they include: 

• A reassessment of the existing hierarchical control structure; 

• Increased, automated data sharing about system conditions over a wider area; and 

• Review of protective relaying practices in the industry. 

For the Midwest area as a whole we need the participation of all major transmission 
systems in an RTO. This will end the prospect of the risks posed by a Swiss cheese 
configuration of systems, some in an RTO and others not. 
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Finally, for the Midwest ISO in particular, acceptance by FERC of our tariff filing to 
establish energy markets in our territory is critical. This will bring added elements of region 
wide action that are not present today - a security constrained generating unit commitment 
program and a real-time security constrained economic dispatch. 

Of the items I mentioned, the first, mandatory reliability standards is largely in the hands 
of Congress. As to the development of more infrastructure in our region, the Midwest ISO 
issued its first transmission expansion plan this June. It calls for construction of SI .3 billion of 
already planned projects. It identifies another S .5 billion of proposed reliability projects. 
Commitment of participating transmission owners to pursue these projects is crucial for the 
future. The cooperation of the states in allowing timely construction is equally critical. 

The remaining items will call for the strong interplay of industry participants and the 
national government mediated through or directed by the Department of Energy and the Federal 
Energy Regulatory Commission. 

This concludes my remarks and I would be pleased to answer questions. 
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SUMMARY 

The New York Independent System Operator (“NYISO”) operates New York’s statewide 
high-voltage electric transmission system. The exact causes of the August 14 th cascading 
blackouts are still unknown and the NYISO will not speculate on them at this time. It is clear, 
however, that New York and Ontario were directly in the path of massive power flows that took 
down major portions of the New York system in seconds. The NYISO is cooperating with the 
International Commission’s investigation and expects the Commission to provide the most 
definitive assessment of what happened. The New York Department of Public Service is also 
conducting an inquiry with which the NYISO is, of course, cooperating. 

The immediate electrical events that caused the blackout in New York occurred at 
4:11 :00 p.m. Within a few seconds, the New York system was hit by onrushing power flows, 
reversals and severe frequency and load oscillations. The transmission system was unable to 
withstand these severe conditions. Approximately 20% of the load, however, continued to 
receive service during the disturbance. Unfortunately, New York City was completely without 
power at this point. 

Immediately after the event, the NYISO began implementation of its restoration plan. 
Statewide service was completely restored by 10:30 p.m. Friday, August 15 th . The restoration 
process followed NYISO’s pre-arranged plan and worked well. Furthermore, preliminary 
analysis indicates that New York systems operated as designed. However, it appears that the 
power swings experienced by New York were beyond what the power system was designed to 
withstand. The complex protective mechanisms installed on New York’s transmission system 
and power plants worked as intended and no significant damage to the infrastructure resulted. 
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Even though it is not yet clear what the cause or causes were of the August 1 4 blackout, it 
is clear that many actions need to be taken to avoid future problems. These actions include: (i) 
making mandatory reliability standards and certain operating protocols set by the North 
American Electric Reliability Council (“NERC”); (ii) making mandatory the incorporation of 
power systems into ISOs and RTOs; (iii) strengthening the transmission grid in New York state 
and surrounding areas; and (iv) enhancing the inter-regional planning process. 
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Testimony of William J. Museler 

Good morning, ladies and gentlemen. My name is William J. Museler, and I am the 
President and Chief Executive Officer of the New York Independent System Operator, or 
NY1SO. I appreciate the opportunity to brief the subcommittee on what we know so far about 
the August 14, 2003 blackout and our restoration operations. My testimony today will focus on 
several important federal policy initiatives for improving electric reliability, an agenda that the 
NYISO and others have been advocating in New York State for several years. 

Immediately prior to coming to the NYISO, I was the Executive Vice President of the 
Transmission/Power Supply Group of the Tennessee Valley Authority, which in terms of MW 
served, is the size of New York. Prior to that, I was Vice President of Electric Operations at 
Long Island Lighting Company. I serve as the Chairman of the ISO/RTO Council, and have 
served on the NERC Board and as Chairman of the Southeast Electric Reliability Council. I am 
a graduate of Pratt Institute and Worcester Polytechnic Institute. 

The NYISO was created to operate New York’s bulk transmission system and administer 
the wholesale electricity markets. We are a New York not-for-profit organization and started 
operation in 1999. As you know, we are pervasively regulated by the Federal Energy Regulatory 
Commission (“FERC”). As provided in the Federal Power Act, we are also regulated with 
respect to certain financings by the New York State Public Service Commission. 

I would like to make clear at the outset the areas that we know and those that we do not. 
While I am, of course, aware of what has been in the press regarding the events that initiated the 
blackout in a significant part of the Eastern Interconnection, 1 am not yet able to tell you 
anything in detail about those events because they have not yet been determined in detail, and 
details in this case are extremely important. Because the initiating events happened in a very 
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short period of time - really just a matter of seconds - and happened away from New York, 
understanding them fully depends largely on interpreting electronic data that we do not have. 

The International Commission formed by President Bush and Prime Minister Jean Chretien of 
Canada is being given the data 1 and is undertaking its interpretation. We are, of course, 
cooperating fully in this investigation. The U.S. end of that investigation is well underway and is 
headed by the Department of Energy. Like you, I’m anxiously awaiting their conclusions. 

Within New York State the NYISO is cooperating with the Department of Public Service in its 
own inquiry into the August 14 events. 

In addition to outside investigations, the NYISO began its own investigation and analysis 
within hours of the event. The NYISO is reviewing its own records to determine the precise 
sequence of events that took down major portions of the New York system within fractions of a 
second. We have, in our preliminary analysis, identified two uncontrollable power swings that 
led to the New York system disturbance that occurred at about 4:11 :00 p.m. 

Up until the event, our system was operating normally, well within applicable criteria and 
with adequate reserves. The immediate electrical events that caused the blackout in New York 
occurred at 4:11 :00 p.m. Within a few seconds, our system was hit by onrushing power flows, 
reversals, and severe frequency and load oscillations. The transmission system was unable to 
withstand these severe conditions. However, several hydro plants in upstate New York, as well 
as the Quebec tie line, remained in service, as did the majority of the upstate transmission 
system. Thus, about 20% of the New York load continued to receive service during the 
disturbance. Unfortunately, New York City was left completely without service at this point. 

1 NERC has been designated as the central data collection and analysis point and all data is being sent to 

them. 
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Immediately after the event, the NYISO began implementing its restoration plan. The 
first step in the restoration process involves stabilizing the system and restoring our ties to the 
neighboring control areas. After that, the process of bringing power plants and outside sources 
back online must take place, including the delicate balancing of the power they can supply with 
the demand in the individual area being restored. If the demand were greater than the supply, the 
system would crash in the affected area, and fortunately that did not occur. 

Within about three hours, we were able to restore one major tie to the remainder of the 
Eastern Interconnection at Ramapo. The first major power plant was returned to service in just 
under an hour after that, and a few minutes later we re-established a transmission path to New 
York City. Throughout the next day, there was a painstaking process of bringing generators 
back to the system and re-energizing lines. Statewide service was completely restored by 10:30 
p.m. Friday, August 15 ,h . The restoration process followed NYISO’s pre-arranged plan and it 
worked well. 

Preliminary analyses indicate that the New York system operated as designed, given the 
event, and that the power swings New York experienced were beyond anything the system had 
been designed to withstand. In an occurrence such as the recent blackout, the greatest danger to 
electric service is potential damage to the system itself— -the power plants and the transmission 
lines. Had that kind of damage occurred, it could have taken days, weeks, or even months to 
restore. Fortunately, the complex protective mechanisms that had been installed on New York’s 
transmission system and on its power plants worked as intended and no serious damage was 
done. This protection shortened the restoration process considerably. 

The Subcommittee has requested our view of “the next steps. . .to ensure that we do not 
have another incident of this nature.” As I stated earlier, however, the events that so affected 
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New York and other states on August 14 th are not yet known in sufficient detail to plan and 
implement specific solutions. Nevertheless, we believe it makes sense to examine the known 
problems that could give rise to other reliability concerns in the future. I will go over for you 
several important policy initiatives for improving electric reliability in which we believe the 
federal government should take a leading role. 

Mandatory Reliability Standards . We believe that the reliability standards set by the 
NERC, which are now voluntary, should be made mandatory. That issue is now before the 
Congress in the energy legislation now before a conference committee. 

Communications . We also believe that the standards should mandate significantly 
improved communications and operating protocols among the various regions of the country, 
since we are now painfully aware of the extent to which events in one region can affect 
neighboring regions. Right now, there is no expectation that a non-adjacent system operator 
would communicate to other, non-contiguous control areas the existence of a condition or 
disturbance on its system or other systems that could jeopardize other regions. While there is no 
guarantee that such improved communications would make possible anticipatory actions that 
would prevent the spread of a problem, it is obvious that advance warning would give operators 
more time to try to take protective actions. 

ISOs and RTOs . We also believe that the incorporation of power systems into ISOs and 
RTOs should be mandatory. The main difference between an ISO and an RTO is its geographic 
scope. An RTO generally covers a larger geographic area than an ISO. Interestingly, the four 
most populous states in the Country have all chosen to operate as single state ISOs or RTOs. 
FERC Order No. 888, which provides the framework for open access to the interstate 
transmission system to facilitate wholesale competition, encouraged but did not require 
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transmission-owing utilities to create ISOs. The decision to create an ISO or other type of entity 
with similar functions, such as an RTO, is currently voluntary and rests with state utility 
commissions. 

ISOs and RTOs generally act as the primary interface between generators, transmission 
owners and other participants in the wholesale electric marketplace. ISOs and RTOs accomplish 
this by dispatching the power system in their control area (he., directing the power plants to 
generate a specific amount of power at a specific time) to supply electricity to customers while 
maintaining safety and reliability. (In addition, ISOs and RTOs generally facilitate and 
administer a number of different electricity markets, thereby providing market participants with 
the ability to sell and purchase various services on an unbundled basis.) Given the extensive and 
growing commerce that takes place in electricity, it is clear that reliability requires coordination 
and operation at a level above that of the local electric company. In today’s environment this can 
best be done by an ISO or RTO. 

Adequate Electric Generation . In addition to the previously stated policy initiatives, 
there are some actions that can be taken to help ensure that other reliability problems do not 
arise. New York has been short of generation in the recent past and projections indicate that 
deficiencies are likely again later this decade. That shortage will grow and will represent both a 
reliability concern and. in our new competitive markets, a cost to consumers. The financing and 
siting of new power plants are issues that must be dealt with through the markets for electricity 
and reasonable siting laws. The NY1SO has already reformed its capacity markets to encourage 
investment in needed facilities and is working with neighboring regions to develop regional 
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Strengthening the Transmission Grid . Adequacy of the transmission grid affects both 
reliability and the cost to consumers. Inadequate transmission hampers free trade and 
competition, resulting in unnecessary cost to consumers. It also enables systems better to 
withstand potentially disruptive events. New York’s transmission grid and its internal planning 
process needs to be strengthened. Current incentives for building transmission are inadequate. 
Likewise, both intra and inter-regional planning processes should be improved, and in the case of 
interstate facilities, a federal override (backstop federal siting authority) may be appropriate. 

In this brief statement, 1 have tried to summarize the state of the investigations into what 
we know about how we handled the recent blackout in New York. I have tried to do so without 
speculating on things about which it is premature to draw conclusions. Needless to say, once the 
results of the international investigation are available, the NYISO will move aggressively to 
implement appropriate changes, as indicated by that investigation. Finally, I have taken the 
opportunity to alert the subcommittee to some of the measures, which can help to avoid future 
problems. 

1 want to thank you for the opportunity to come here today, and assure you that we will 
be cooperating with the on-going inquiries into the outage. 
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Mr. Chairman, my name is Jim Kerr. { am a member of the North Carolina Utilities 
Commission (NCUC), having served on that body for a little more than two years. I am also a 
member of the Electricity Committee of the National Association of Regulatory Utility 
Commissioners (NARUC), the Immediate Past President of the Southeastern Association of 
Regulatory Utility Commissioners (SEARUC) and currently serve, along with Marilyn 
Showalter, Chair of the Washington Utilities and Transportation Commission, as the Co-Chair of 
the Alliance of State Leaders Protecting Electricity Consumers (Alliance!. 

Please note that in submitting this testimony, I am speaking for myself, not the NCUC, 
NARUC, the Alliance, or SEARUC. Although my comments are informed by my discussions 
and exchanges with fellow regulators and industry stakeholders, these are my views, not those of 
the aforementioned organizations or their members. 

I very much appreciate the opportunity to appear before the Senate Subcommittee on 
Oversight of Government Management, the Federal Workforce and the District of Columbia and 
to assist in your consideration of the cause of the August 2003 Northeast blackouts and the 
appropriate response of the federal government to ensure that similar events are not a part of our 
nation's future. Specifically, 1 have been asked to discuss the causes of the blackouts, as well as 
the current federal role in managing and regulating the generation and the transmission of 
electricity, i te discreet issues that you are addressing today are of great national importance; 
they also are relevant to broader, equally important, subjects facing this Congress in the context 
of pending federal energy legislation. 

As you consider these matters, I encourage you to carefully consider the impact of any 
proposed electricity legislation on each region of the country, including the Southeast, because of 
the significantly different manner in which electric service is provided to retail customers in each 
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part of the country. Thus far in its consideration of the recent blackout. Congress has heard only 
from federal officials or from representatives of the regions of the country that were affected 
directly by the blackouts. I hope to provide you with a broader perspective from which to 
consider both the recent events and the appropriate federal responses. To this end, I would like 
to describe for you the structure of the electric system in North Carolina and the Southeast, our 
approach to reliability, and my thoughts on appropriate federal responses to the recent blackouts. 

At the outset I want to make clear that my purpose is not to tout the electric system in 
North Carolina (or the Southeast) as perfect or the only acceptable system design. Nor do 1 
advocate congressional action that would prevent a region of the country from pursuing the 
electric market structure that best suits that region’s needs. My message is that Congress must be 
careful that, in responding to recent events, it does not pursue perceived solutions that will 
adversely affect electric systems that are, in fact, working well. 

OVERVIEW OF THE ELECTRIC SYSTEM IN THE SOUTHEAST 

Regulatory Jurisdiction 

The NCUC, like other similar bodies across the country, is an agency of state government 
responsible for regulating the rates charged and terms and conditions of retail electric service 
provided by the entities defined by our General Assembly as “public utilities.” Under North 
Carolina law, our electric jurisdiction extends to “persons” owning and operating equipment and 
facilities for the production, generation, transmission, distribution, and furnishing of electricity. 
Our statutory authority, where it applies, is broad and plenary, encompassing all aspects of the 
retail service provided by the utilities under our jurisdiction. 1 


! The NCUC’s jurisdiction does not. however, extend to rural electric cooperatives and municipal distribution 
systems, subject to certain limited exceptions. 
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While the NCUC’s jurisdiction is focused on the provision of retail electric service, the 
Federal Energy Regulatory Commission (FERC) also plays a role. In 1935, in response to the 
United States Supreme Court’s Attleboro 2 decision. Congress enacted Title II of the Federal 
Power Act, which created FERC (then the Federal Power Commission). Congress' intent was to 
establish a body that could regulate certain interstate activities of utilities that the Court in 
Attleboro suggested were beyond the reach of state regulators. Accordingly, FERC was given 
authority over wholesale power sales and the transmission of electricity in interstate commerce. 3 
This combination of the states’ plenary authority over retail electric service and FERC’s 
interstitial jurisdiction over specific matters that might evade state regulation has been the 
cornerstone of the regulatory framework in the Southeast for almost 80 years. 

The Provision of Electric Service in the Southeast 

Electric service in the Southeast continues to be provided, in large part, by vertically- 
integrated utilities subject to the regulatory oversight of state commissions. These utilities own 
and operate generation, transmission and distribution facilities, which they use to serve their 
customers, including hospitals, schools, churches, and homes. Although I have not made a 
comprehensive study of the laws in other Southeastern states, North Carolina law' clearly 
contemplates the continued existence of such vertical integration. The only common exceptions 
to this model in most of the Southeast are rural cooperatives or municipal electric systems, some 
of which own electric distribution systems, but not generation or transmission facilities. 


2 Public Utilities Commiss io n v. Attleboro Steam & Co .. 273 US 83, 71 L Ed 549, 47 S Ct. 294 (1927). 

5 16 USC $824. et sea. (2003). 
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The Role of the Wholesale Power Market in the Southeast 

The continued existence of the traditional industry structure throughout most of the 
Southeast does not mean that we are indifferent to the potential benefits of a properly- 
functioning wholesale market. On the contrary, the NCUC recognizes that a properly- 
functioning wholesale market can benefit the retail customers of our vertically-integrated utilities 
in a number of ways. First, the wholesale market can provide enhanced opportunities for our 
utilities to procure competitive generation fromjndependent power producers as an alternative to 
utility-built options. Secondly, the wholesale market can provide opportunities for additional 
short-term economy purchases, allowing our utilities to reduce their costs by purchasing power 
instead of operating more expensive units on their own systems. Finally, the wholesale market 
can allow vertically-integrated utilities to share reserves, effectively reducing the costs of 
maintaining system reliability. As a result, 1 do not believe that any of my colleagues disputes 
the benefits of a properly-functioning wholesale market to retail customers despite the continued 
presence of traditional, vertically-integrated utilities. 

The NCUC. and the utilities we regulate, have taken steps to take advantage of the 
potential benefits of the wholesale market in recent years. When the utilities procure additional 
capacity to meet anticipated future load, they typically issue a request for proposals to solicit 
wholesale power offers that are compared with the cost of self-build options prior to making a 
final resource procurement decision. Also, the records in our fuel adjustment cases demonstrate 
that our jurisdictional utilities purchase substantial amounts of power from marketers and brokers 
in lieu of generating power from their own facilities when it is economic to do so. The NCUC 
adopted procedures to facilitate the recovery' of the costs associated with such purchases in order 
to encourage our utilities to make the best economic decisions for retail customers and we have 
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revised our generating plant certification rules to make it easier to site and construct merchant 
generating facilities in our state. To date, we have not rejected any application for the issuance 
of a merchant plant certificate. Thus, North Carolina Utilities Commission has embraced the 
opportunities for cost savings and reliability improvements available on the wholesale market 
and I feel confident that the rest of the states in the Southeast have taken steps to do the same. 

However, the potential benefits of wholesale market improvements for the retail markets 
in North Carolina and most other Southeastern states are not unlimited. The ultimate purpose of 
the wholesale electric market is the same as most wholesale markets — supporting the retail 
market. As noted above, the vast majority of the power sold at retail in the Southeast is 
generated by utility-owned facilities. Although certain municipal and cooperative electric 
systems rely more heavily on the wholesale market, the simple fact of the matter is that, for the 
foreseeable future, the impact of wholesale market improvements in the Southeast is likely to be 
relatively limited. While the importance of the wholesale market in the Southeast may increase 
over time, the potential benefits of an improved regional wholesale market in the near term 
should not be oversold. As a result, any attempt to redesign wholesale electric markets should 
include a careful weighing of the costs and benefits for the retail markets, including the costs of 
implementation. 

Conclusion 

With the exception of Virginia, retail competition is not authorized anywhere in the 
Southeast, and it appears that our neighbors in Virginia are reconsidering their movement toward 
retail competition. Arkansas recently repealed the retail competition statute that it enacted a 
number of years ago. It is my impression from talking with colleagues throughout the region 
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that none of the other Southeastern states are likely to move to retail competition in the near 
future. 

At this point, the general perception among Southeastern regulators is that the regional 
system for providing electric service is, on balance, working well. Our rates are among the 
lowest in the country. We have not experienced any significant reliability problems in recent 
years. Our reserve margins generally are adequate. A study of the regional transmission 
infrastructure performed by SEARUC found no material transmission bottlenecks. 4 While our 
electric system is not perfect, the available evidence has not led our state legislatures to support 
radical restructuring of the type adopted in certain other parts of the country. Unquestionably, 
the decision of whether, when, or how to restructure retail markets is a decision for each state to 
make instead of a matter to be decided at the federal level. As a result, the existing industry- 
structure in the Southeast is likely to remain in place for the foreseeable future. 

THE SOUTHEAST’S APPROACH TO RELIABILITY 

The states and electric utilities in the Southeast were not directly affected by the August 
14, 2003, blackouts. Therefore, we do not have any firsthand knowledge of the specific causes 
or contributing events which led to the blackouts. Nevertheless, even with limited and imperfect 
knowledge at this time, we have an obligation to try to assess this series of events and leam from 
them. Thorough reviews are underway by the affected states, Congress, United States 
Department of Energy, the North American Electric Reliability Council (NERC), and others that 
should shed more light on this event. Utilities and state commissions in the Southeast are 
closely analyzing and studying the results of these and other investigations to determine what 

4 See Letter from James Y, Ken- II to Chairman Pat Wood. Federal Energy Regulatory- Commission, dated August 
22, 2002, attached hereto as Exhibit A. 
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“lessons can be learned” and whether new measures should be adopted in our jurisdictions to 
further reduce the possibility of similar events affecting our region in the future. I would add that 
both North Carolina and South Carolina utility regulators have already met with the Southeastern 
Electric Reliability Council (SERC), PJM, and the utilities that we regulate in order to begin this 
process. 5 

Although we do not yet have clear answers, we as regulators must be able to answer the 
questions of what happened on August 14' h and how-we prevent a reoccurrence. Indeed, those 
questions are the reason we are all gathered in this room today. While the exact causes of the 
blackouts are still unknown, numerous industry experts have narrowed their focus to at least 
three general factors as potential causes: 

1. Accountability for Reliability; 

2. Transmission Planning and Investment; and. 

3. Operational Coordination and Communication. 

I would like to take the next few moments to describe how the Southeast and. in particular, my 
state of North Carolina addresses each of these factors. 

Accountability for Reliability 

- In the aftermath of the blackouts, all of our utilities were asked whether similar problems 
could occur in the Southeast. For the utilities that are in unrestructured systems, the answer was 
easy. We know who has the statutory responsibility for both generation and transmission 
adequacy. Accidents can happen, but there is political and regulatory accountability in the 
utilities and in their regulators. In restructured systems, it is harder to find both the technical and 
political accountability. A large fraction of the grid is used for strictly commercial transactions, 
rather than bundled sales to native load. Utilities, and by extension their state and local 


See Exhibit D for the presentation made by North Carolina utilities to the NCUC. 
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regulators, have a smaller role in the building of both new generation and new transmission 
which makes accountability divided and unclear. 

Accountability and responsibility are clear for North Carolina and much of the Southeast. 
North Carolina law requires public utilities to provide reliable and adequate electric service to all 
customers in their assigned territories at reasonable rates. Annually, these utilities provide 
reports and resource plans to the NCUC in order to demonstrate the steps they are taking to 
fulfill those obligations in the near term and in the long term. These reports and plans are subject 
to public scrutiny by customers and regulators. Each exhibit is exposed to public debate and the 
results are resource plans that are responsive to the needs of customers while meeting the 
utilities' statutory obligations. Further, utilities in North Carolina are subject to answering for 
service deficiencies in the form of formal and informal customer complaints lodged with the 
NCUC. Even on its own motion, our Commission can inquire into any aspect of the utilities' 
operations and take steps to require improvements to transmission, distribution and generation 
service or take definitive rate-making action if circumstances so require. In short, there is never 
any question what party has responsibility for reliability in resources and planning. Nor is there 
any doubt about the process by which those goals are achieved. 

Transmission Planning and Investment 

The planning process in the Southeast is a bottom up approach that has been in place for 
many years. Each utility in North Carolina and many other utilities in the Southeast are mandated 
to engage in integrated resource planning, which involves the joint planning of generation, 
transmission and demand management. The utilities are then required to file these integrated 
resource plans with the state commissions in order to assure them that reliability will be 
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maintained. The process begins with long term (typically 10 year) plans developed by the 
individual utility using NERC, SERC and system reliability criteria. These plans are then 
combined at the sub-regional level (for North Carolina it is called VACAR - Virginia, North 
Carolina and South Carolina) and are tested using computer models against NERC reliability 
criteria and for the ability to transfer power between systems. The plans from each sub-region 
are further combined to develop a SERC regional plan (which encompasses VACAR. Southern. 
Entergy and TV A). Again, the plans are tested using computer models for compliance with 
NERC and SERC reliability criteria. Finally the plans are combined at the multi-regional level 
where they again are tested for reliability compliance. This process has worked very well and 
continues to keep the lights on in the Southeast. 6 

The overall success of the integrated and regional planing processes in the Southeast 
stands in sharp contrast to the claims of some who say that America has a "third world 
transmission grid.” I would agree with my colleague Dr. Schriber of the Ohio Public Utilities 
Commission that these statements are simply untrue, at least for the Southeast. I do not doubt 
that some areas of the country need more transmission construction. The Southeast, however, is 
not lacking in transmission investment. In 2002. the utilities in the Southeast invested over $1 
Billion in the transmission grid and over the next five years the utilities plan to invest more than 
$6 Billion in the Southeast transmission grid. 

More importantly, the Southeast transmission grid is operating well for the specific 
purpose for which it was designed - to serve local load with adequate reserves for reliability and 
wholesale transactions. We have not experienced the blackouts, market meltdowns, price spikes, 

6 For a more detailed description of the SERC planning process, see Letter to Representative Tauzin from SERC 
Executive Director William Reinke, dated August 28, 2003, and the SERC presentation to the North Carolina 
Utilities Commission, dated August 26, 2003, attached hereto as Exhibits B and C respectively. 
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and various problems that have plagued other regions. Even more minor transmission problems, 
which are referred to by the industry as Transmission Loading Relief measures (TLR), have not 
cropped up as often in the Southeast as they have elsewhere. For example, throughout North and 
South Carolina, only 3 TLRs were called in 2002. Whereas, PJM called 95 TLRs and the 
Midwest ISO called 950 TLRs in 2002. 1 Can the system planning processes used in the 
Southeast be improved? Certainly, there is always room for improvement. However, the 
planning, construction and operation of the transmission grid in the Southeast has served the 
region well to date. 

Operational Coordination and Communication 

The now well-publicized transcripts of conversations between RTO and utility personnel 
leading up to the blackouts suggests that lack of clear, prompt communication may have 
contributed to the problems. On this point, it is worth reiterating that whatever theoretical value 
there might be in disaggregating utility operations, there is always going to be the practical 
concern that too much disaggregation is going to require lightning fast communication AND 
reaction by multiple parties, each with their own limited perspectives and interests. Can this be 
accomplished seamlessly? Perhaps. But what we know for sure is that the structure of vertical 
integration, coupled with regional coordination, allows utilities in the Southeast to respond to 
unforeseen events quickly and efficiently. The ability of utilities that operate transmission and 
generation facilities to coordinate and adjust those operations instantaneously in real time 
unquestionably enhances their ability to react to emergencies such as the events of August 14'\ 


7 See www. nerc.co m/ pub/svs/all up dl/ oc/scs / loes/trends.htm for a more detailed description of TLRs throughout the 
nation. 
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In my opinion this approach is responsible for the consistent reliability the Southeast enjoys in 
normal and extraordinary conditions. 


APPROPRIA TE FEDERAL RESPONSE 


In your consideration of the appropriate federal response to the events of August 14 th , I 
would encourage you to first consider a few very basic principles. First, acknowledgeJhe one 
lesson from the events of August 14 ,h that is beyond question - - the provision of reliable 
electricity to the nation is of vital importance to the lives of its citizens and to its economy and 
security. Second, until more is known about the actual cause or causes of the blackouts, you 
should be very careful about either jumping to conclusions or using those events as the basis for 
any legislative action. Finally, any potential solution you might consider must “do no harm” to 
the existing industry structure as it might exist throughout the country. With these basic 
principles in mind, I would like to comment first on some of the broader policy initiatives that 
some proponents have raised as possible solutions to the issue of reliability and then on several 
of the more discreet proposals that more directly impact on reliability. 

Regional Transmission Organizations and Standard Market Design 

Some proponents of the FERC’s restructuring efforts are pushing mandatory Regional 
Transmission Organizations (“RTOs”) and Standard Market Design (SMD) as the cures for the 
yet-to-be-determined causes of the blackouts. 1 do not believe that such expansive overhauls are 
necessarily relevant and they are certainly not the answer. As noted above, the blackouts 
occurred in areas that have gone the farthest in implementing RTOs and SMD. The formation of 
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RTOs and the adoption of SMD do not add a single transmission line or a single kilowatt of new 
generation capacity, but will cost many millions of dollars to implement. RTOs and SMD may 
be helpful in some regions, and any area that desires to pursue such restructuring efforts should 
be allowed to do so. By the same token, RTOs and SMD do not appear to be necessary or 
beneficial for every part of the country. In that regard, I think it is safe to say that most of my 
fellow regulators in the Southeast have considerable doubts about the appropriateness of those 
policies for our region. Southeastern regulators have been considering these issues generally and 
in the context of the specific RTOs that have been proposed in the Southeast (i.e., SeTrans, 
GridSouth and GridFlorida). Also, SEARUC recently commissioned a cost-benefit analysis to 
determine if the benefits of RTOs and SMD outweighed the costs. 8 The results of that study 
raise serious questions as to whether the benefits of forming an RTO and implementing SMD in 
this region would exceed the costs and risks. A more recent cost-benefit analysis performed by 
the Department of Energy raises similar questions about whether the implementation of RTOs 
and SMD would increase costs to retail customers in the Southeast. 9 Moreover, even though 
DOE’s study suggests that there may be net savings from the implementation of SMD under 
optimal conditions; those savings are extremely modest - less than 1% on a nationwide basis 
and take years to materialize. This strikes me as a very thin potential return for such a high-risk 
investment. 


8 See http://www.state.va.us/scc/searuc/cra_study.pdf for a copy of the Benefits and Cost of Regional Transmission 
Organizations and Standard Market Design in the Southeast dated November 6. 2002. 

9 See www.gn ergv.gov for a copy of the Department of Energy Standard Market Design Cost/ Benefit Report dated 
April 30, 2003. 
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Furthermore. I am concerned that RTOs and SMD could have the unintended effect of 
harming reliability. FERC’s policy initiatives appear to be moving towards disaggregation (the 
separation of generation and transmission functions) and this raises the question as to whether 
such separation has an effect on reliability. Among other things, these FERC initiatives 
encourage long distance transfers of power. Because the ability to serve load becomes more 
susceptible to problems caused by the loss of critical transmission lines, these long distance 
transfers raise reliability concerns. In contrast, when generation is located near load, there is less 
distance for the power to travel before it reaches the load and hence less opportunity for 
problems. 

I am also concerned that the FERC’s restructuring model usurps state jurisdiction over 
electric service, which would seriously impede the state commissions’ ability to exercise their 
statutory responsibility and to assure that retail customers in their states are served reliably and in 
a cost effective manner. This set of circumstances leaves state regulators with little authority vis- 
a-vis the RTO to address day-to-day issues much less to deal w'ith extraordinary events such as 
the recent blackouts. While the FERC has indicated that state commissions might be able to play 
an advisory role in the new world of SMD and RTOs. this is a poor substitute for jurisdiction and 
direct accountability and gives Southeastern states serious concern. 

Reliability Standards 

A consensus is building within the electric industry to support federal legislation to 
establish mandatory, enforceable reliability standards through an industry-led. self-regulating 
organization (i.e., NERC). I support that effort; in particular I support the current electric 
reliability language in the House Energy Bill (HR 6, Section 216). NERC has developed an 
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appropriate set of planning and operating standards and those standards will most likely be 
further improved following the evaluation of the August 14 th blackouts. While voluntary 
compliance with these standards has worked well in the Southeast, it is possible that some 
regions may need mandatory reliability rules to provide additional assurance of reliability. 

As part of the effort to establish mandatory reliability standards, some have proposed that 
the FERC be authorized to review and enforce NERC’s reliability standards. While 1 recognize 
that some governmental authority probably needs to have oversight responsibility, 1 am 
concerned about such a proposal for two reasons. First, the FERC has no reliability expertise 
that I am aware of; thus, FERC would need to rely upon industry experts such as NERC who 
have traditionally been the ones in charge of reliability criteria. Second, the FERC must not be 
allowed to use any newfound reliability authority as a way to promote its regulatory agenda to 
restructure the wholesale market. For this reason, any federal legislation would need to be 
carefully structured to ensure that the FERC’s additional authority is limited to the promotion of 
reliability. Legislation must not allow FERC to use reliability authority to pursue the mandatory 
restructuring of markets across the nation. In particular, the FERC must not be allowed to use 
any such reliability legislation to try to force mandatory RTO participation or the adoption of 
SMD. 


Transmission Incentives 

To encourage transmission construction, some entities are recommending the adoption of 
incentive rates. In certain circumstances, incentive rates might be appropriate. For example, the 
adoption of accelerated depreciation for transmission construction would appear to be an 
appropriate catalyst to spur new investment. However, 1 do not believe as a general matter that 
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incentive rates are always necessary, at least not in the Southeast. As previously discussed, the 
utilities in the Southeast, with oversight from their state regulators, have done and continue to do 
an admirable job of ensuring that sufficient transmission investment is made to provide 
economical and reliable electric service to consumers. Importantly, this investment is being 
made under the traditional regulatory model of relying upon vertically-integrated utilities that 
receive a regulated return on their investment. Incentive rates should be directed at transmission 
projects that would not be constructed under the traditional model; otherwise such incentives will 
serve only to raise rates for consumers. 

Another transmission-related incentive being discussed involves the tax treatment of 
transfers of transmission assets. Generally speaking, this provision would limit the tax exposure 
of utilities that transfer transmission assets to other entities. This provision may be desirable to 
remove an obstacle for utilities that are interested in transferring their transmission assets; it must 
be emphasized, however, that in many, if not most, states in the Southeast, any such transfer 
would require the approval of the state regulatory commission. 

Backstop Siting Authorin’ 

Another proposal being discussed in response to the blackouts is-to provide the FERC 
siting authority to allow it to authorize the construction of new transmission investment. This 
proposal raises serious concerns. As an initial matter, sufficient transmission lines have been and 
are being constructed to maintain reliable service to consumers in the Southeast. While some 
claim that problems arise when utilities try to build interstate transmission lines, it bears noting 
that many utilities in the Southeast are multi-state in nature and already work with different 
jurisdictions in planning and siting transmission facilities. In short, the siting of transmission 
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lines, like most land use issues, raises many local concerns. If a transmission line is not 
authorized by the state, there would likely be very good reasons for that decision - reasons that 
should not be easily discounted. For these reasons, any legislation to give the FERC “backstop" 
authority to site transmission must acknowledge the states’ primary role in the siting of 
transmission and carefully prescribe the FERC’s authority to overrule those decisions. This is 
especially true given the various efforts that are already underway to encourage greater 
coordination and cooperation among the- states on planning and siting issues. 

The above mentioned statements on federal siting authority are my own opinions but, as a 
member of NARUC, I feel obligated to state that NARUC respectfully opposes any provisions 
that contemplate federal siting authority for transmission (direct or backstop). NARUC believes 
that states should retain authority to site electric facilities. Congress should support the states’ 
authority to negotiate and enter into cooperative agreements or compacts with federal agencies 
and other states to facilitate the siting and construction of electric transmission facilities as well 
as to consider alternative solutions to such facilities, such as distributed generation and energy 
efficiency. 

CONCLUSION 

The actual cause or causes of the August 1 4 lh blackouts are as of yet unknown. What is 
certain is that this event provided all of us with a dramatic and unfortunate reminder of the vital 
importance of reliable electricity to all of our lives. As this Congress begins to understand better 
the actual causes of the blackouts and the appropriate responses it might take, I encourage you to 
keep the unpleasant memories of that day in the forefront of your minds as a reminder of how 
much we all risk in formulating our responses. Much is being done correctly and appropriately 
in all of the regions of the country and you should first “do no harm" to that w'hich is working 
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effectively in the various regions. Only then should you undertake responses that will build upon 
that which is working in order to enhance the reliability of electric sendee to the entire nation. 

The Southeast’s model for utility service is not the only such model that can work, and 
my testimony here today should not be interpreted as indicating otherwise. However, there can 
be no serious debate that the electric system in our region works and works well. It is a system 
that our citizens have invested in for decades and it has delivered on its purpose - - producing 
highly reliable, reasonably priced power that meets the expectatians and needs of our citizens. 
While not perfect, the system of regulation upon which our electric system is based is in large 
part responsible for this result. With that said. I understand that other regions of the country 
have problems that they are trying to solve and 1 support the Southeast helping in any way that 
we can, as long as that support is not a detriment to the reliable and efficient electric system we 
have built for the Southeast. We continue to believe that the balance of risks and tradeoffs 
associate with changing this system are best assessed by the policymakers closest to, and 
politically accountable for, the actual operations of the system. Accordingly, in order to ensure 
the continued provision of reliable service in our region, this Congress and federal regulators 
must avoid the precipitous implementation of policies which will disrupt the smooth functioning 
of our current system. By the same token. Congress and federal regulators can and should 
implement policies that are narrowly drawn, cost effective, reflective of the regional differences 
that exist, and which can improve upon the reliability and operation our electric system and that 
of other regions. 

Thank you for the opportunity to be with you today and to provide my perspective on 
these important issues. 
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On behalf of the Southeastern Association of Regulatoi 
Utility Commissioners (SEARUC), I would like to take th 
opportunity to respond to the transmission infrastructure testimor 
which you presented to the United States Senate Committee c 
Energy and Natural Resources on July 24, 2002. There are sever 
items within your testimony that do not reflect the current situatic 
in the southeastern states, and for that reason we feel compelled i 
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The grid in the southeast ... is inadequate to serve tt 
needs of the competitive wholesale market. 

[I]ncumbent transmission companies have tended to act 
ways that favor their own generation and impede pow 
flows for independent generators. 

The central question to be resolved in the southeast is, wt 
should pay for the new transmission facilities that a. 
desperately needed for the region as a whole? 
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The Honorable Pat Wood, III 
August 22, 2002 
Page 2 

• Although the residents of Mississippi, Alabama and Louisiana are benefiting from 
the investment dollars, jobs and tax benefits of [merchant] power plants, they are 
reluctant to pay for any new transmission lines that may be needed to enable 
these plants to reach their intended interstate markets. 

(Testimony at 7.) 

First and foremost, we would note that there are not now, nor are there projected 
to be, any transmission infrastructure deficiencies in the southeastern region of the 
United States. In fact, during the Federal Energy Regulatory Commission (FERC) 
infrastructure conference in Orlando, Florida, over which you presided during May of 
this year, we presented you with SEARUC’s in-depth analysis of the transmission 
situation in the southeast, the Southeastern Infrastructure Assessment ( SIA Report ) 1 , 
based on up-to-date data as reported by the North American Electric Reliability Council 
(NERC) and the Southeastern Electric Reliability Council (SERC). Transcript of the 
Southeast Infrastructure Conference, at 169 (FERC Docket No. AD02-1 3-000, May 9, 
2002) (Ir.). This analysis indicates that the southeast has sufficient transmission 
capacity for the present and foreseeable future as well as a significant projected excess 
of generation capacity. ( SIA Report at 1.) Upon receiving this information in May, you 
and your fellow commissioners expressed great satisfaction with the current state of 
affairs in transmission infrastructure in the southeast, particularly when compared with 
that of other regions of the country. 2 

Secondly, contrary to the statement in your Senate testimony, we are not aware 
of any actions by the regulated incumbent transmission companies in the southeast to 
“impede power flows" or otherwise discriminate against independent generators. In fact, 
the only testimony contradicting the adequacy of the southeastern transmission 
infrastructure that was presented during the infrastructure conference in Orlando was 
made by a member of your staff, Mr. Scott Miller. His testimony, however, was based 
only on "anecdotal" evidence rather than actual data, as was noted for the record with 
concern by Commissioner Brownell. (Tr. at 19.) In addition, your recently issued notice 
of proposed rulemaking on standard market design (SMD NOPR, Docket No. RM01-12) 
offers scant evidence of discriminatory conduct by any utility in the southeast, hardly 
sufficient to justify the proposed assumption of federal jurisdiction over bundled retail 
service in the region. We have previously offered our assistance in dealing with any 


' In addition to providing the SEARUC Southeastern Infrastructure Assessment to you and your staff in 
person in Orlando, it is available on the Internet at <www.psc.state.fl.us/general/publications/searuc.pdf>. 

2 MR. CALLAHAN: Well, do you feel good about what you've heard? 

MR. WOOD: I do. 

MR. CALLAHAN: Do you feel good about the infrastructure in the southeast? 

MR. WOOD: For me, yeah. Linda's nodding. Nora? i saw some of the issues and 1 haven't had time to 
digest them about the, in the transmission study yesterday that was released by the Department of 
Energy that I'd like to, and I think probably will mention a little bit in the [SEARUC] study. And since I just 
got both of those today, I'd like to say a qualified yes. But I like what 1 heard. I think it's, kind of makes it 
easy to move onto the next one [infrastructure conference], which won't probably be as easy. (Tr. at 200.) 
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such anti-competitive behavior and ask, again, that you provide us with any facts or 
complaints supporting such allegations. 

Thirdly, after having received our report in Orlando on the adequacy of the 
transmission infrastructure in the southeast and having expressed satisfaction with what 
you heard, you reported to Congress that “new transmission facilities ... are desperately 
needed for the region.” However, several corporate officers of the major transmission- 
owning electric utilities in the southeastern states testified in Orlando that transmission 
infrastructure has been and is being built as needed to meet the needs of consumers in 
the southeast. Mr. Bill Newman, Senior Vice President for Transmission and Planning in 
Operations at Southern Company Services, testified that traditional planning has 
worked and is working well in the southeast to ensure that transmission is planned and 
expanded appropriately. He also testified that these transmission systems were 
designed and built so there is no congestion even with a transmission line or generating 
plant out of service. With respect to new infrastructure, Southern has budgeted to invest 
$3 billion in transmission over the next five years to serve its native load and its current 
network and point-to-point wholesale customers. (Tr. at 76-77.) In summary, our 
regulated electric utilities have invested, and are continuing to invest, in transmission 
infrastructure as needed for customers in the southeast, and the region does not 
“desperately” need new transmission facilities as stated in your testimony. 

We agree, however, that one vitally important issue is that of who will pay for the 
transmission expansions and upgrades that will be necessary to move to market the 
tremendous amount of merchant plant capacity that has been proposed to be built in the 
southeast. While some of this excess capacity can be sold in the southeast, it will not be 
economic for those plants to supply power solely in the southeast and stay in business. 
Because the developers of these merchant plants chose to locate in the southeast while 
planning to sell to other parts of the country, a tremendous amount of new transmission 
infrastructure will have to be added and dedicated to that purpose. It has been and 
remains our contention that, if our ratepayers will not benefit from the excess generation 
capacity that moves to distant markets, they should not be burdened with the billions of 
dollars in new transmission costs needed to export that capacity. We are pleased that 
the FERC agreed in the SMD NOPR to consider participant funding, and would 
encourage you to apply such a pricing policy to fund the transmission requirements of 
such plants. 3 


3 Participant funding has been endorsed previously by SEARUC and by the National Association of 
Regulatory Utility Commissioners: 

[NJow therefore be it 

RESOLVED, That the Board of Directors of the National Association of Regulatory Utility 
Commissioners (NARUC), convened in its February 2002 Winter Meetings in 
Washington, D.C., hereby urges the Federal Energy Regulatory Commission (FERC) to 
utilize a pricing policy which provides that the cost of investments, that have been 
demonstrated, through an even-handed assessment of transmission, generation and 
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Lastly, we would inform you that states such as Mississippi, Alabama and 
Louisiana have not, contrary to the assumptions set forth in your testimony, received 
either employment or tax revenue benefits commensurate with the billions of dollars that 
will be required for new transmission infrastructure to transmit to other regions of the 
country the excess merchant power described above. These merchant plants, 
meanwhile, are consuming the natural gas and water resources of those states and are 
imposing other costs that must be balanced against any benefits. In short, there is 
certainly no level of benefit to those states that could come close to offsetting the 
tremendous level of transmission costs that would adversely impact their ratepayers 
and regional economic development efforts if the FERC adopts a socialized pricing 
regime for the corresponding level of new transmission investment that would be 
required to create the FERC's desired “interstate” highway system for electric 
transmission. 

The various electric industry initiatives currently being pursued by the FERC 
raise challenging issues that do not need to be further confused by failures to articulate 
the actual differences between and among the various regions of the country. 
Representations that attempt to treat all areas of the country, or all state commissions, 
as one group, are inaccurate, frustrate any progress toward true dialogue, and inhibit 
the thoughtful consideration of actual problems and potential, appropriate solutions. The 
very real differences among the regions must be respected, considered carefully, and 
articulated by all involved parties for real understanding to occur and appropriate 
decisions to be made. 

We appreciate the opportunity to clarify your testimony to the Senate Committee. 
You have undertaken an aggressive agenda at the FERC, and we appreciate this, and 
other opportunities, to participate in the dialogue. 


efficiency alternatives, to be needed to maintain the reliability of the existing transmission 
system is recoverable through rates paid by ail transmission customers; and be it further 

RESOLVED, That the cost of upgrades and expansions that are necessary to support 
incremental new loads or demands on the transmission system are borne by those 
causing the upgrade or expansion to be undertaken except that the FERC should not 
preclude the assignment of interconnection cost to the general body of ratepayers within 
a State when that State's regulatory body determines that such allocation is in the public 
interest. 


NARUC Resolution On Transmission Pricing Policy, Adopted by the NARUC Board of Directors at the 
February 2002 Winter Meetings in Washington, D.C. 
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With best wishes and warm personal regards, I am 

Sincerely yours, 



President 


cc: The Honorable Linda K. Breathitt, Commissioner 

The Honorable Nora Mead Brownell, Commissioner 
The Honorable William L. Massey, Commissioner 
Southeastern United States Senators 
Southeastern Governors 

Members, United States Senate Committee on Energy and Natural Resources 
Member Commissions, Southeastern Association of Regulatory Utility 
Commissioners 
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Southeastern Electric Reliability Council 

600 North Eighteenth Street P.O. Box 2641 Birmingham, Alabama 35291 
Telephone: (205) 257-6407 Fax:(205)257-0408 
Web Page: www.sercl.org 


William F. Reinke 

Executive Director 


Via Facsimile 


August 28, 2003 


The Honorable Billy Tauzin 
United States House of Representatives 
2183 Rayburn House Office Building 
Washington, D. C. 20515-1803 

Dear Representative Tauzin: 

On behalf of the members of the Southeastern Electric Reliability Council I am pleased 
to respond to the questions concerning the events of August 14 posed in your August 
20, 2003 letter. Please advise if you will require additional information from SERC 
concerning this event. 


(1 ) What were the basic causes and contributing events that led to the August 14 ,h 
blackout and its severity? Describe the following in your answer: 

a) The location, character, and proximate cause of the initial disruption in the 
transmission and supply of electricity: and 

b) The “cascading” effect of the disruption through multiple utility systems 
and States. 

The Southeastern Electric Reliability Council (SERC) has no unique 
information concerning this disturbance. The bulk electric systems in SERC 
remained intact during this event. 

(2) What efforts have been taken to secure the supply, transmission, and distribution 
since the blackouts of 1965 and 1977 in the Northeast, and why were these 
efforts apparently inadequate to prevent the blackout or otherwise minimize the 
area affected? What efforts have been taken in other parts of the country to 
prevent blackouts and how effective have these efforts been in preventing or 
minimizing blackouts? 

SERC has no information as to why efforts were apparently inadequate to 
prevent the blackout or otherwise minimize the area affected. 


William K. Newman, Chairman 
Southern Company Services, Inc. 
600 N. 18th Street, P.O. Box 2641 
Birmingham, AL 35291-8200 
(205)257-6462 FAX (205) 257- 


Terry Boston, Vice-Chairman 
Tennessee Valley Authority 
1101 Market Street MR 3 H 
Chattanooga, TN 37403-2801 
(423)751-6000 FAX (423) 751- 
sw 


Charles While, Secretary-Treasurer 
South Carolina Electric & Gas 
1426 Main Street 
Columbia, SC 29201 
(803) 217-9518 FAX (803) 933- 

7T4? 
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Efforts that have been taken in SERC to prevent events described include: 
Creation of SERC in 1970. “SERC’s purpose is to further augment the 
reliability and adequacy of bulk power supply in the areas served by the 
member systems. To this end the Council will: 

a) promote the development of reliability and adequacy 
arrangements within the region; 

b) participate in the establishment of reliability policies, 
standards, principles and guides; 

c) participate in the measurement of performance relative to 
these policies, standards, principles, and guides; 

d) ensure conformance to and compliance with these policies, 
standards, principles, and guides; 

e) develop and exchange information with respect to planning 
and operating matters relating to the reliability and adequacy 
of bulk power supplies.; 

f) review as necessary activities within the region on reliability 
and adequacy in order to meet expected standards and 
measurements; 

g) provide a mechanism to resolve disputes on reliability issues 
in a manner that meets the needs of the parties and the 
region; 

h) provide information with respect to matters considered by 
the Council, where appropriate, to the Federal Energy 
Regulatory Commission (FERC) and to other federal and 
state agencies concerned with reliability and adequacy; 

i) take such actions as are necessary to adapt and put in place 
the Regional Compliance and Enforcement Program 
("RCEP”) contemplated by the April 25, 2001, “Agreement 
for Regional Compliance and Enforcement Program” 
between SERC and the North American Electric Reliability 
Council. Such action shall include, but not be limited to, 
preparing and entering into voluntary contracts with SERC 
Members establishing and governing RCEP standards and 
other matters, and adopting sanctions and processes for 
imposition of sanctions for noncompliance with RCEP 
standards.” 1 

SERC Members have developed and implemented systems to protect the 
integrity of the bulk electric system for underfrequency events. Thirty 
percent or more of the connected load in SERC is subject to automatic 


1 Southeastern Electric Reliability Council Agreement dated January 14,1 970, amended October 23, 2002 
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underfrequency load shedding. 

SERC has formally integrated Operations, Engineering, Markets, and 
Critical Infrastructure Protection in its committee structure. 

SERC systems have integrated technological changes that enhance 
system reliability including: 

• wide spread use of remote sensing technology 

• implementation of System Control and Data Acquisition systems 

• on line reliability tools including load flow and state estimator 
software 

SERC systems have been aggressive in pursuit of improved planning and 
operations coordination including: 

• Addition of a Conferencing Center to facilitate reliability: hotline 
calls; daily Reliability Coordinator calls; other member 
communications. 

• Annual Emergency Plan Coordination Seminar 

o In preparation for the 2003 Emergency Plan Coordination 
Seminar SERC members’ Under Frequency Load Shed 
information was gathered, consolidated, and distributed for 
discussion. 

• Annual System Operator Conferences 

o Recognized and applauded by NERC 

• Data bases for Transmission and Generation Outage Coordination 

• Regional procedures and plans: emergency assistance 
procedures; black start regional restoration plan 

• Participation in numerous and ongoing intra-regional and inter- 
regional analyses of projected operating conditions and long-term 
assessments of the reliability of the transmission system. SERC 
was a leader in first implementing inter- and intra-regional studies 
and analyses. 

• SERC was a leader in developing and maintaining NERC Certified 
System Operators 

• A SERC representative introduced the tagging concept to NERC, a 
fundamental tool widely used for congestion management. 

SERC systems have also put in place an extensive program to assure 
compliance with NERC Operating Guides and Planning Standards. SERC 
audits control room operations in the Region on a three-year cycle to 
ensure that member systems comply with all applicable NERC standards. 
SERC has begun an audit process for compliance with NERC Planning 
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Standards with the intent of auditing its members on a three-year cycle for 
compliance with NERC Planning Standards. 

• In 2002 SERC made more thanl 100 assessments of control room 
operations and found these systems to be 99.4% compliant with 
NERC Standards. None of the non-compliance findings affected 
or jeopardized the operation of the bulk electric system in SERC. 

• Although the SERC Planning Standards Compliance Effort is less 
than five years old, it is more extensive than that recommended by 
NERC. Unlike operations, the NERC Planning Standards do not 
require quarterly or monthly assessments. Therefore, typically 
fewer planning assessments are performed. We are encouraged 
by the compliance findings to date and expect that as this effort 
matures SERC systems compliance with Planning Standards will 
exceed the 99+% target established by the operations groups. 

3) What equipment, measures, or procedures worked as intended on August 
14 th to prevent even greater disruption to the supply of electricity, to prevent 
greater damage to the generation and transmission system, and to bring 
generation back on line after the disruption? 

SERC system facilities all worked as intended on August 14. Any 
abnormalities observed on SERC system facilities as a result of the event 
were well within design parameters. 

4) How can the nation's electrical system, including both transmission capacity 
and reliability, be improved to prevent a recurrence of the events of August 
14 >h ? Please identify what measures may need to be taken by all involved in 
the governmental and nongovernmental sectors. 

SERC systems have developed a robust transmission system with more 
than 100 transmission connections to its neighbors to the north and west. 
SERC systems have experienced weather-related service disruptions, 
e.g., ice storms, tornadoes, hurricanes, but have not experienced an event 
similar to that of August 14. 


Very truly yours, 


William F. Reinke 
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Cons umer Federation of America 



THE FEDERAL RESPONSE TO THE 2003 BLACKOUT: 
TIME TO PUT THE PUBLIC INTEREST FIRST 

STATEMENT OF DR. MARK N. COOPER 


Subcommittee On Oversight Of Government Management, 
The Federal Workforce And The District Of Columbia 
Committee On Governmental Affairs, United States Senate 

September 10, 2003 


Mr. Chairman and Members of the Committee, 


My name is Dr. Mark Cooper, i am Director of Research at the Consumer Federation of 
America (CFA)! I also appear today on behalf of Consumers Union. 1 2 We have been deeply 
engaged in the debate over electricity restructuring and deregulation for almost two decades. I 
have submitted to you a list of appearances I have made before Congress and Federal 
Agencies, as well as state regulatory commissions, on this issue. I have also submitted the 


; have conducted 


since 1997, soon after the first radical restructuring laws were passed in a couple of states. 
Every six months for the last twenty years we have been cautioning policymakers not to 
experiment with electricity or treat it like any other commodity. 


I greatly appreciate the opportunity to appear before you today to present the residential 
ratepayer view of the federal role in the ongoing troubles of deregulated electricity markets. It is 
about time that the voice of the little guy and gal, the people who pay the bill, is heard on this 
matter. It is about time that you get the perspective of local jurisdictions that have had the good 
sense not to go down the road of electricity restructuring and deregulation or have decided to 
change course after being badly burned by deregulation and restructuring. Two-thirds of the 
states have figured out that deregulation is a road to ruin. It is time for federal authorities to 
change course too, or at least to pause for a substantial period while they rebuild the physical 
and institutional infrastructure of the electricity gird. 


Electricity Restructuring Does Not Add Up for Residential Consumers 


In the wake of the massive blackout in the Northeast, government officials and industry 
experts are calling for a massive upgrade of the transmission system that will cost between $50 
billion and $100 billion. The annual carrying costs for such a capital outlay are certain to be in 
the range of $10 billion to $25 billion, or even more if the merchant model being pushed by the 


1 CFA is the nation's largest consumer advocacy group, a non-profit association of 300 pro-consumer groups, with a 
combined membership of 50 million, founded in 1968 to advance the consumer interest through advocacy and education. 

2 Consumers Union is a nonprofit membership organization chartered in 1 936 under the laws of the State of New York to 
provide consumers with information, education and counsel about goods, services, health, and personal finance; and to initiate and 
cooperate with individual and group efforts to maintain and enhance the quality of life for consumers. Consumers Union's income is 
solely derived from the sale of Consumer Reports, its other publications and from noncommercial contributions, grants and fees. In 
addition to reports on Consumers Union's own product testing, Consumer Reports, with approximately 4.5 million paid subscribers, 
regularly carries articles on health, product safety, marketplace economics and legislative, judicial and regulatory actions which 
affect consumer welfare. Consumers Union's publications carry no advertising and receive no commercial support. 
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Federal Energy Regulatory Commission (FERC) is pursued. Many experts are beginning to 
admit that a substantial part of the upgrade costs are caused by the need to support the 
increased strains on the system that occurs in deregulated electricity markets. In contrast to 
these huge costs, the Department of Energy conducted a study earlier this year using 
extraordinarily optimistic and unfounded assumptions but still projected less than $1 billion of 
efficiency gains from implementation of FERC’s Standard Market Design. FERC’s own study 
found equally meager gains, while studies by other regulatory commissions question whether 
even those small benefits are realistic. 

A consumer does not need a degree in electrical engineering to see that these numbers 
do not add up. If the costs outweigh the benefits, why should we bother? At a minimum, 
policymakers should inquire as to what it would cost to run a reliable system without the added 
demands on the system and associated costs of supporting deregulated markets. 

Frankly, with the discovery of massive infrastructure costs needed to support 
deregulation and the run up in natural gas prices, there is no chance that deregulation will 
produce benefits for the vast majority of residential consumers. In the northeast and mid-west 
where regulators did a miserable job of protecting consumer interests in the 1980s and 1990s, it 
was possible to use regulatory mechanisms to hold consumers harmless during the early phase 
of deregulation, but those days are gone. Consumers now face huge price increases as the 
dash to gas-fired generation in the past decade will flow through to the electricity meter and 
meet up with the excessive capital costs of deregulated markets. 

Electricity Is A Unique and Vital Service, Not Just A Commodity 

This is not the first time that the electricity market has thrown a curve at deregulation. In 
fact, restructured electricity markets have lurched from crisis to crisis - the price spikes of 1998, 
the outages of 1999, and the California meltdown of 2000-2001 . All of these events share a 
common cause - electricity is different from any other service or commodity. 

Historically, the uniquely American approach to delivering this vital and difficult service 
was to allow private companies to own both transmission and generation and provide service in 
exclusive territories, subject to public interest obligations. The integration of generation and 
production fostered coordination and effective management of the network. Exclusive territories 
lowered the risk and costs associated with long-term inflexible assets. Public interest 
obligations, such as the obligation to serve all customers at just and reasonable rates, protected 
the public from the abuse of monopoly power while preserving companies' incentive to invest in 
the network. 

This pragmatic approach was certainly not perfect, but it achieved a critical balance 
between public and private interests. In the past decade, policymakers lost sight of these 
fundamentals and deregulation upset that balance, particularly for the transmission system. De- 
integration quickly turned into disintegration. 

Electricity has no substitutes. It is not storable. It is essential to health, safety and the 
economy. It must be delivered under incredibly demanding conditions through an extremely 
capital intensive infrastructure. 

It was blatantly irresponsible for Federal and state authorities to rush ahead with 
deregulation without the necessary physical and institutional infrastructure to support the 
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markets they were trying to create. It would be grossly negligent for Congress to allow 
restructuring to continue without taking a long pause to repair the damage that has been done 
to consumers and the electricity infrastructure. 

Restructuring and Deregulation Make It More Difficult to Ensure Reliability 

Make no mistake about it; deregulation and restructuring have increased the stress on 
the transmission system. There are numerous economic and operational mechanisms through 
which electricity restructuring and deregulation increased pressures on the nation's electricity 
transmission network: 

• A dramatic increase in the number and complexity of transactions, which the 
system was not designed to support. 

• Difficulties of coordination and planning as competition and contracts replace 
vertically integrated operational and administrative decisions. 

• Deregulation short-circuited utility incentives to invest in transmission because 
the private interests of facility owners came into conflict with the shared, public 
nature of the transmission grid and created a disincentive to spend on 
maintenance because of profit pressures and the perceived competitive 
disadvantage associated with spending on a system shared with potential 
competitors. 

• Increasing needs for excess capacity to cope with market manipulation problems 
that plague electricity markets and to dampen price spikes that result from trying 
to treat electricity like a commodity, all of which must be paid for with the higher 
cost of merchant finance. 

• Failure to account for the social and environmental constraints on increasing 
transmission capacity and to provide a framework for comprehensive planning 
that integrates alternative approaches, like energy efficiency and local 
(distributed) generation (such as co-generation). 

Given the massive costs of deregulated markets that are now coming into view and the 
meager gains that such markets appear to promise, not to mention a track record of market 
manipulation, price volatility and lack of consumer choice, it may be a lot cheaper for the handful 
of states who have deregulated to go back than to force the majority of states down the problem 
riddled road toward deregulation. 

The Grid Is Basic Infrastructure: A Highway, Not A Market 

T ransmission facilities are critical infrastructure of a modern society and digital economy 
that must be dedicated to promoting the public interest. They are part of a shared system in 
which the fate of each user and producer is tied to the behavior of others. The fundamental 
problem with transmission is not inadequate economic incentives to invest; utilities were willing 
to do so before deregulation the problem is public resistance to the building of additional 
transmission lines for environmental, health and safety reasons. The social cost of transmission 
facilities is far greater than their economic costs. For this reason, scarcity of transmission in the 
economic sense is likely to be a permanent part of the industry landscape. 
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Moreover, the benefits of shared transmission facilities that support the overall network 
are difficult to align with private calculations of costs and benefits. The problem is both 
geographic, determining which benefits accrue to which areas, and intergenerational, 
recognizing that different parts of the system may benefit differently from the same investment 
across time. Today’s investment to serve a long distance transaction may be a core part of 
tomorrow's system serving native (local) load. The shared nature of facilities makes it more 
difficult for private investors to recover their costs and to overcome the social resistance to the 
siting of facilities. The shared nature of facilities across jurisdictions makes it more difficult to 
reconcile competing interests. Such public investment is best carried out within the framework 
of a comprehensive plan. Yet, integrated resource planning is harder to implement in the 
deregulated model, if it is not abandoned altogether. 

We take the primary lesson of the decade of deregulation to be that we need to restore 
the balance of public and private interests in the electricity sector. Society cannot rely on 
private actors to ensure that adequate investments are made in vital public goods, such as the 
electric transmission grid. The transmission system is a highway, not a market, and should be 
developed under a public interest model in which the primary purpose of all participants is to 
ensure reliability and protect the public. The obligation to serve, which transmission utilities 
properly bear, must be matched with a duty to build. Bribing merchants to provide these vital 
public goods, such as through "incentive" payments, unbridled expansion into non-utility 
businesses, and the auctioning of transmission capacity to the highest bidder, will be particularly 
expensive. 

These lessons have been clear for quite some time. Federal authorities simply seem 
unwilling to get the message. Two years ago, in the midst of the last crisis of electricity 
restructuring and deregulation, in testimony I entitled “The Federal Role in the Deregulation 
Tragedy," 1 offered the following conclusions about transmission: 

The failure to recognize the important role of the continuing monopoly in 
transmission resulted in the under-regulation of the wires segments of the 
industry. The transmission wires are the highways of commerce over which 
electricity flows. This is a highway system, not a market, which constitutes an 
essential, bottleneck facility with virtually no redundancy and never likely to 
support head-to-head competition. One of its primary inputs is right-of-way, 
which relies on governmental power of condemnation. The biggest obstacle to 
the expansion of transmission capacity is a social externality - public concern 
about ugly wires and local health effects - not inadequate economic incentives. 
Proposals to let the marketplace solve the wires problem are not likely to 
succeed, since given the market power that the wire "owner” would possess and 
the non-market barriers to expanding capacity, profit maximization would only 
result in the abuse of market power and the creation of artificial scarcity rents. 

The right model for transmission is a public or private entity imbued with the 
public interest and dedicated to ensuring that this essential facility fulfils its public 
functions - ensuring reliability and supporting nondiscriminatory market 
transactions in the widest area possible to achieve economies of coordination 
and maximum competitive effect. It must be independent of market participants 
and directly accountable to public authorities for achieving those goals. 

Transactions must be standardized and transparent, with the creation of an 
exchange in which all rates terms and conditions can be identified. Brokers must 
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be subject to rules that are similar to those applied to financial transactions like 
stock sales. 

We offered similar advice to the Congress last year in a report entitled "All Pain; No 

Gain:” 


Rather than rushing ahead with restructuring and deregulation, Congress and 
FERC need to step back and fully understand the implications of the abuses, 
operational disruptions, and the financial crisis that has occurred in the electricity 
industry. Congress must restore simplicity and transparency to the industry. The 
first goal must be to reinforce consumer and investor protections. A 
comprehensive review of the national transmission system should be conducted. 
Effective mechanisms for planning and expanding the grid should be 
demonstrated in reality. Institutions for managing the grid and overseeing trading 
should be transfigured before moving forward. 

I could go back two decades and the message would be the same. I understand the 
pressures to do something in the wake of the blackout, but when it comes to electricity, doing 
just anything will not help. You have to do the right thing, or you will make matters worse. 

What Federal Authorities Should Not Do 


Policymakers could have eased the transition to competHive generation markets by 
recognizing the physical and institutional infrastructure that would be needed to support greater 
competition, but they did not. Perhaps they realized that presenting a true picture of the 
difficulty of electricity deregulation would have made it impossible to sell to the public. Whatever 
the reason behind the underestimation of the difficulties of deregulation, the build-up of 
problems now makes the implementation of competition a much riskier proposition. Not only 
has the inadequacy of institutions and facilities grown, but public confidence in the process has 
also been eroded. Congress needs to start solving the problem by stopping the deregulation 
train. 


Do not repeal the Public Utilities Holding Company Act (PUHCA). Congress does 
not need to allow utilities to diversify into non-utility businesses and form huge multi-state 
holding companies by repealing PUHCA to solve the reliability problem. This would subject the 
utility industry to less oversight, by allowing utilities to play a shell game with their assets and 
increasing FERC's responsibility, which has so far been completely unable to deal with the 
manipulation of markets in the west and with the misreporting of energy prices. 

Do not allow the FERC to impose its complex ‘‘Standard Market Design” on the 

nation. Regional transmission organizations that are dominated by industry and preempt local 
accountability while forcing utilities into markets that allocate transmission resources to the 
highest bidder, with no assurances that transmission is presently adequate or that additional 
transmission capacity would be built or adequately maintained, are a prescription for disaster. 

Do not rely on industry self-regulation for reliability. The proposal to move from 
voluntary self-regulation to mandatory self-regulation misses the point. The problem is not the 
voluntary part; it is the self-regulation part. The industry will simply not regulate itself 
sufficiently, especially in a market-oriented system, to protect the public. The private interests of 
the large players will always come first. 
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Do not create private transmission monopolies. Transmission services are a natural 
monopoly and part of a shared network. Transferring control to unregulated private parties will 
simply allow them to abuse captive customers and shift costs onto the backs of ratepayers 
throughout the system. 

What Federal Authorities Should Do 

Federal authorities should devote all of their energy to promoting the public interest, not 
the profits of merchant generators and transmission owners, by studying, strengthening and 
managing the interstate transmission system. Any interstate transmission organization 
must be based on fairness and public accountability. We must create new institutions that 
can reconcile the interests of the states and include representation of consumer interests. 
Interstate compacts or federal state-joint boards are a possibility. 

Fairness requires that an interstate transmission organization embody a process for fair 
representation of all interests affected by transmission projects. Local consumers and citizens 
must not be excluded from the process. Accountability demands that local officials who get 
the phone calls when the lights go out must be in a position of authority. Standards must be set 
by responsible authorities and be mandatory, with stiff penalties for failure to comply. Industry 
self-regulation will not do. Public accountability also requires transparency. The competition 
between the FERC and the DOE, and the army of private consultants muddying the picture of 
what happens on the transmission grid, is unacceptable. 

The obstacle to expanding transmission is not inadequate economic incentives; the 
obstacles are environmental, public health, and safety concerns. Even if economic incentives 
were a problem, the solution is not to increase incentives; it is to lower risk. The cost of 
bottleneck, infrastructure facilities are much lower when they are funded through a utility finance 
model. Utility investments in transmission facilities will easily attract capital if policy makers 
restore their traditional quality of stable, dividend paying investments. 

Congress should require a framework for comprehensive planning that considers all 
alternatives. It should get serious about energy efficiency, like mandating higher minimum 
standards for air conditioners, which would reduce demands on the grid at its most vulnerable 
times, hot summer days. It could also give a boost to local (distributed) generation, which 
has the double benefit of adding generation resources to the system while not using long 
distance transmission lines, whose failure triggered the recent blackout. 

Unfortunately, both the House and Senate bills that are being reconciled in conference 
violate virtually every one of these consumer "Do's” and “Don'ts.” The fact that the Congress 
has failed to act in the past several years is actually a good thing for consumers because 
Congress has never once come close to passing legislation that would do the right thing. Now 
is the time to focus on the real problem, restore accountability and oversight over the industry 
and put the public interest first. 
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Testimony of Pat Wood, III 
Chairman, Federal Energy Regulatory Commission 
Before the Subcommittee on Oversight of Government Management, 
the Federal Workforce, and the District of Columbia, 

Committee on Governmental Affairs 
United States Senate 
November 20, 2003 

Mr. Chairman and Members of the Subcommittee: 

Thank you for the opportunity to appear before you today to discuss the findings 
and recommendations of the interim report of the joint U.S.-Canada Task Force on the 
August 14, 2003 Northeast blackouts. 

Watching and studying this blackout has been a sobering experience. The 
reliability of the North American electric system is normally so excellent that this year’s 
notable service interruptions - from the August 14 blackout in the Northeast, blackouts in 
London, Italy, Argentina, Norway, and elsewhere, and recently from Hurricane Isabel - 
have forced us all to look afresh at all our old assumptions about the value of reliable 
electric service and what it takes to keep the lights on. 

Here’s what I have learned from the blackout investigation and this interim report, 
and from thinking about those other blackouts this year: 

The blackouts in the Northeast, Italy, London and elsewhere have a common 
theme - something routine happens, like a tree contacting a powerline or a minor relay 
setting done wrong, and the time to react and keep the system stable suddenly shrinks 
beyond the capability of human control, when the machines take over. The grid is a 
tremendously complex system, and the interconnectedness that allows us to benefit from 
higher reliability and lower costs also causes the domino failures experienced in many 
countries in recent months. We cannot ever prevent blackouts, but we can and must learn 
to reduce their frequency, magnitude and impact. 

The best way to manage blackouts is to prevent them, not to hope for heroic 
rescues when we’re already in a jam. The secret to reliability lies in making sure that 
every transmission owner, control area operator, and reliability coordinator takes care of 
the basics - adequate tree trimming, adequate training for emergency as well as routine 
operations, effective communications within and across organizations, and having 
effective back-up facilities, procedures and tools. The investigation clearly shows that 
had FirstEnergy trimmed its trees, used a solid state estimator program after the trip of 
the Eastlake 5 unit and regularly throughout the afternoon of August 14, and trained its 
operators better to recognize and deal with emergencies, the blackout would not have 
happened. 
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The blackout study shows that current reliability standards were violated by 
FirstEnergy and the Midwest Independent System Operator. We need better compliance 
and tough, clear standards. The FERC will be working closely with NERC and the 
stakeholders to develop those standards and to implement the reliability provisions of the 
energy bill if Congress approves it. 

We do need some major investments in new transmission facilities and new grid 
technologies, especially those that make it easier for us to manage the basics. But we 
need to make those investments wisely, for lines and equipment that expand the 
reliability parameters of the grid where it is needed - for instance, new sources of 
reactive power for the Cleveland- Akron area - appear to be long-overdue. Further 
analysis conducted by the blackout investigation teams will teach us much about how the 
cascade spread and why it stopped where it did, and that will help us design a system that 
over the longer term should perform more reliably and cascade more narrowly. The new 
energy bill offers new options to site long-needed transmission lines and to pay for 
reliability investments, and I am eager to put those measures in-place. 

We also need to invest in hardware and software that let operators manage the grid 
more effectively. Tools that improve system monitoring, evaluation, visibility, 
visualization and information sharing about grid conditions over a wide region will allow 
operators to manage the grid more reliably on a day-to-day basis as well as in 
emergencies. Our colleagues at the Department of Energy have done some excellent 
work in this area over the past few years and we will be looking to these technologies and 
others to raise the bar for grid management capabilities. 

Transmission is regulated at the federal and state level. Clearly we need to 
regulate it better, to assure the reliability that Americans deserve. As the present energy 
bill recognizes, the days of voluntary reliability standards with no enforcement teeth must 
end. Federal regulators must work closely with our state colleagues to make sure that 
utility cost-cutting that allows 14 inch diameter trees to grow in transmission rights-of- 
way, or inadequate operator training, or the widespread use of inadequate software 
ineffectively used, must end. I pledge that my commission will work closely with our 
colleagues in Ohio and other states to deliver better regulation for better reliability. 

Some claim that electric competition and higher energy flows caused under- 
investment in an over-worked grid and made this blackout inevitable. What they ignore 
is that the operator’s primary charge is to work the system you’ve got, and that the 
operator has the power to cut back any transaction, tighten the operational limits on any 
line or power plant, and even cut customer load, if that’s what it takes to keep the system 
safe and secure. Markets do not compromise reliability, but we must redouble our efforts 
to assure that all necessary reliability measures are taken. 


2 



195 


Perhaps the saddest portion of the blackout report is Chapter 6, the comparison of 
the August 14 outage to other major outages in North America. The common factors are 
overwhelming: 

• Conductor contact with trees due to inadequate vegetation management 

• Insufficient reactive power 

• Inability of system operators or coordinators to recognize and understand 
events across the broad regional system 

• Failure to ensure that system operation was within safe limits 

• Lack of coordination on system protection 

• Failure to identify emergency conditions 

• Ineffective communication 

• Lack of “safety nets”, and 

• Inadequate training of operations personnel. 

The seven outages reviewed span from 1965 through 1999. Extensive analysis followed 
each outage, and blue ribbon panels developed good recommendations after each. Some 
of the recommendations that followed that outage have been implemented, but not many. 
It is my hope that with the adoption of the new reliability provisions of the energy bill, 
we can finally implement most of those recommendations and stop making the same 
mistakes over and over. The cost of those mistakes is too high, and our nation and our 
people deserve better. 

The cost of major blackouts is immense, in human and financial costs. New 
transmission facilities and tools - AC and DC lines, substations, capacitors, sensors, state 
estimators, visualization programs, and others - are not cheap. Business practices that 
improve transmission reliability - like thorough tree-trimming, operator training, and 
development of procedures and plans for routine and emergency communications - are 
not cheap either, and will cost more on customers’ bills. But if you ask the New Yorkers 
who were stuck in the subway on August 14, or the Cleveland residents who had to boil 
their water for days, or the folks around Maryland and Virginia who sat without power 
for as long as a week after Hurricane Isabel, most will tell you that they’d rather pay a 
little more for electricity if it will reduce the number, length and impacts of such outages. 
The cost of reliability improvement is trivial in comparison to the costs and consequences 
of poor reliability, and we need to be less penny-wise and pound-foolish when we do this 
math. 

Thank you. I will be pleased to answer any questions you may have. 
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Testimony of James W. Glotfelty 
Director, Office of Electric Transmission and Distribution 
U.S. Department of Energy 

Before the 

U.S. Senate Committee on Governmental Affairs 
Subcommittee on Oversight of Government Management, the Federal Workforce, 
and the District of Columbia 
Washington, D.C. 

November 20, 2003 


Mr. Chairman and Members of the Subcommittee: 

Thank you for the opportunity to testify here today and outline the findings of the 
U.S.- Canada Power System Outage Task Force investigating the blackout on 
August 14, 2003. 

Three months ago today, large sections of the United States and Canada were still 
recovering from one of the largest power blackouts in our nation’s history. Since 
the blackout, hundreds of technical experts have worked tirelessly to help the U.S- 
Canada Task Force determine how and why it occurred. 

Yesterday, the Task Force released an Interim Report that marks our progress to 
date in the search for answers about what happened that day. 

This Interim Report focuses on the events, actions, failures, and conditions that led 
to the blackout and caused it to cascade over such a large region, as well as 
questions relating to nuclear power operations during the blackout and to the 
security of the grid control systems 

It presents facts collected by the investigation team and does not offer speculative 
or unconfirmed information or hypotheses. 

Without going though a line-by-line review of how the system failed that day, I 
would like to walk you through the three causes of this blackout. 

Before I do this, I want to make clear that it is the control area operators who have 
the primary responsibility to maintain system reliability, regardless of conditions. 
They are required to have the tools to ensure the grid is reliable. There were three 
groups of causes: 
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Group 1 - FirstEnergy didn’t properly assess changing conditions on its system, 
which degraded as the afternoon progressed. In particular: 

• FE didn’t ensure the security of its transmission system because it didn’t use 
an effective contingency analysis tool routinely. 

• FE lost its system monitoring alarms and lacked procedures to identify that 
failure. 

• After efforts to fix those alarms, FE didn’t check to see if the repairs had 
worked. 

• FE didn’t have effective backup monitoring tools to help operators 
understand system conditions after their main monitoring and alarm tools 
failed. 

Group 2 - FE failed to adequately trim trees in its transmission rights-of-way. 

• Overgrown trees under FE transmission lines caused the first three FE 345 
kV line failures. 

These lines tripped when contacting trees tharhad grown past their 
maximum allowable limits in their rights of way. 

• Trees found in FE right-of-way areas are not a new problem 

o The investigation found one tree over 42’ tall; one 14 years old; 
another 14 inches in diameter were found in FE rights-of-way. 

o There also was extensive evidence of prior tree-line contacts. 

Group 3 - Reliability Coordinators did not provide adequate diagnostic support to 
assist FE in responding to problems. 

• MISO’s state estimator failed because of a data error. 

• MISO’s flowgate monitoring tool didn’t have real-time line information to 
detect growing overloads. (The flowgate tool was still under development at 
the time of the blackout.) 

• MISO operators couldn’t easily link breaker status to line status to 
understand changing conditions. (The EMS tool was still under development 
at the time of the blackout.) 

• PJM and MISO lacked joint procedures and wide grid visibility to 
coordinate problems affecting their common boundaries. 

According to NERC, these failures violate a number of reliability standards. 
Specifically, 

• FE violated at least four NERC reliability standards. 

• MISO violated at least two standards. 
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A critical reference point in this investigation is 3:05 p.m. At that time, the 
investigations extensive modeling determined that the system was capable of being 
operated reliably. That fact alone eliminates a number of possibilities as causes of 
the blackout. They include: 

• High power flows to Canada, 

• System frequency variations, 

• Low voltages earlier in the day or prior days, 

• Low reactive power output from IPP’s, and 

• Existing outages of individual generators or transmission lines that had 
occurred well in advance of the blackout. 

Finally, the Task Force report finds that: 

• Procedures at the nuclear plants were followed and worked well on 
August 14 th . 

• The nuclear plants all shut down safely when they detected a disturbance, 

• And were restarted safely when the grid was restored. 

• In addition, no deliberate damage or tampering has been found in any 
equipment in affected areas of the grid. 

• And no computer viruses or any sort of illicit cyber activities have been 
identified as factors. 

In closing. Phase One of our Task Force investigation, and the public’s response to 
it, will give us a wealth of information that will be the basis for fonnulating 
recommendations on ways to make our electric system stronger. 

Phase Two of this investigation will include three public forums in Cleveland, New 
York City and Toronto. These public forums will offer an opportunity to all of 
those listed in this report, as well as other interested parties, to provide the Task 
Force with comments and recommendations. 

The Task Force will then issue a final report containing our recommendations for 
improving the electric system and for any appropriate follow-up 

Thank you Mr. Chairman. I will be happy to answer any questions you may have. 
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Summary of the major events that occurred 
on August 14, 2003 

• 12:15 - Eastern Daylight Time (EDT) - inaccurate input data rendered 
MISO’s state estimator (a system monitoring tool) ineffective. 

• 13:3 1 EDT - FE’s Eastlake 5 generation unit tripped and shut down 
automatically. 

• 14: 14 EDT - the alarm and logging system in FE’s control room failed and 
was not restored until after the blackout. 

• 15:05 EDT - 3 of FE’s 345-kV transmission lines began tripping out because 
the lines were contacting overgrown trees within the lines’ right-of-way 
areas. 

• 15:46 EDT - FE, MISO and neighboring utilities had begun to realize that 
the FE system was in jeopardy. The only way that the blackout might have 
been averted would have been to drop at least 1 ,500 to 2,500 MW of load 
around Cleveland and Akron — and at this time the amount of load reduction 
required was increasing rapidly. No such effort was made. 

• 15:46 EDT - the loss of key FE 345-kV lines in northern Ohio caused its 
underlying network of 138-kV lines to begin to fail, leading, in turn, to the 
loss of FE’s Sammis-Star 345-kV line. 

• 16:06 EDT - the Sammis-Star line tripped and triggered the cascade by 
shutting down the 345-kV path into northern Ohio from eastern Ohio. 
Although the area around Akron, Ohio was already blacked out due to 
earlier events, most of northern Ohio remained interconnected and electricity 
demand was high. The loss of the heavily overloaded Sammis-Star line 
instantly created major and unsustainable burdens on lines in adjacent areas, 
and the cascade spread rapidly as lines and generating units automatically 
took themselves out of service to avoid physical damage. 
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Hearing Before the United States Senate 
Committee on Governmental Affairs 

Subcommittee on Oversight of Government Management, the Federal Workforce, 
and the District of Columbia 
November 20, 2003 

Prepared Testimony of 

Michehl R. Gent 

President and Chief Executive Officer 
North American Electric Reliability Council 


Good morning, Mr. Chairman and members of the Committee. My name is Michehl 
Gent and I am President and Chief Executive Officer of the North American Electric Reliability 
Council (NERC). Thank you for inviting me to provide NERC’s perspective on the interim 
report of the U.S.-Canada Power System Outage Task Force on the causes of the blackout on 
August 14, 2003. 

NERC is a not-for-profit organization formed after the Northeast blackout in 1 965 to 
promote the reliability of the bulk electric systems that serve North America. NERC’s mission is 
to ensure that the bulk electric system in North America is reliable, adequate, and secure. NERC 
works with all segments of the electric industry as well as electricity consumers and regulators to 
set and encourage compliance with rules for the planning and operation of reliable electric 
systems. NERC comprises ten Regional Reliability Councils that account for virtually all the 
electricity supplied in the United States, Canada, and a portion of Baja California Norte, Mexico. 

NERC has been an integral part of the joint fact-finding investigation that led to the 
interim report on the August 14 blackout that the U.S.-Canada Power System Outage Task Force 
issued yesterday. NERC fully supports the findings and conclusions in the interim report. With 
respect to what happened on August 14, the key findings and conclusions are detailed on page 23 
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of the interim report, as follows: “inadequate situational awareness at First Energy Corporation,” 
“First Energy failed to manage adequately tree growth in its transmission rights-of-way,” and 
“failure of the interconnected grid’s reliability organizations to provide effective diagnostic 
support.” 

Immediately after the onset of the blackout on August 14, 2003, NERC began assembling 
a team of the best technical experts in North America to investigate exactly what happened and 
why. Every human and data resource we have requested of the industry has been provided, and 
experts covering every aspect of the problem have been volunteered from across the United 
-States and Canada. In the week following the blackout, NERC and representatives of DOE and 
the Federal Energy Regulatory Commission (“FERC”) established a joint fact-finding 
investigation. All members of the team, regardless of their affiliation, have worked side by side 
to help correlate and understand the massive amounts of data that we have received. We have 
had hundreds of volunteers from organizations all across North America involved in the 
investigation so far. 

To lead the NERC effort, we established a strong steering group of the industry’s best, 
executive-level experts from systems not directly involved in the cascading grid failure. The 
steering group scope and members are described in Attachment A. 

On October 15, NERC sent a letter to the CEOs of all reliability coordinators and control 
areas in North America directing them to verify within 60 days that their organizations are 
measuring up to reliability requirements in six key areas: Voltage and reactive management, 
reliability communications, failures of system monitoring and control functions, emergency 
action plans, training for emergencies, and vegetation management. The intent of this action was 

NERC Testimony 

Subcommittee on Oversight of Government Management, the Federal Workforce, and the 
District of Columbia 
November 20, 2003 


Page 2 
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to minimize the likelihood of another blackout in the near future while the investigation is 
ongoing and a lull set of recommendations is being developed. Responses are due on December 
15. The full text of that letter is in Attachment B. 

Chapter 6 of the interim report compares the August 14 blackout to other major 
disturbances on the interconnected bulk electric system. That comparison reveals that some of 
the causes of the August 14 blackout (inadequate vegetation management, failure to ensure 
operation within secure limits, failure to identify emergency conditions and communicate that 
status to neighboring systems, inadequate operator training, and inadequate regional-scale 
visibility over the power system) were repeats from the earlier outages, -but it also revealed some 
causes not seen before (inadequate interregional visibility over the power system, dysfunction of 
a control area's SCADA/EMS system, and a lack of adequate backup to that system). The 
electricity industry has made great strides in responding to the recommendations from those 
earlier investigations, in the form of better communication capabilities, operator certification 
program, better tools for dealing with congestion on the grid, but clearly more needs to be done. 
For one thing, we are now using the bulk electric system harder than we have in the past and in 
ways for which it wasn’t designed. Actions and practices that sufficed when the system had 
plenty of margin for error simply are inadequate when the system is being pushed more to its 
limits as electricity markets are increasingly characterized by larger transactions over greater 
distances. For another, the reliability responsibility for a given area that used to be concentrated 
within a single, vertically integrated organization is, in many parts of the country, now divided 
among several different entities. 


NERC Testimony 

Subcommittee on Oversight of Government Management, the Federal Workforce, and the 
District of Columbia 
November 20, 2003 
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One important step Congress can and should take to strengthen the reliability of the bulk 
power system in general would be to pass legislation to make the reliability rules mandatory and 
enforceable. For several years, NERC and a broad coalition of industry, government, and 
customer groups have been supporting legislation that would authorize creation of an industry- 
led self-regulatory organization, subject to oversight by FERC within the United States, to set 
and enforce reliability rules for the bulk electric system. NERC has developed a world-class set 
of planning and operating standards, though I expect we will find it necessary to improve those 
standards, based on the events of August 1 4. However, as long as compliance with these 
standards remains voluntary, we will fall short of providing the greatest possible assurance of 
reliability that could be achieved through mandatory verification of compliance and the ability to 
impose penalties and sanctions for non-compliance. On Tuesday, the House of Representatives 
passed H.R. 6, a comprehensive energy bill that includes the needed reliability legislation, H.R. 
6 is now before the Senate for action. 

As for the August 14 outage, much remains to be done. As the entity responsible for 
reliability standards for the bulk electric system, NERC must understand and communicate to its 
members what happened on August 14 and why it happened. The interim report is a major step 
in accomplishing that task. NERC must also determine whether any of its standards were 
violated and whether its standards and procedures require modifications to take into account the 
ways in which the bulk electric system is being used. Finally, NERC must assure that measures 
necessary to avoid a recurrence of the August 14 outage are taken. 


NERC Testimony 

Subcommittee on Oversight of Government Management, the Federal Workforce, and the 
District of Columbia 
November 20, 2003 
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NERC will continue to work with the U.S.-Canada Task Force as the investigation 
continues and recommendations are developed. We expect to learn many additional lessons 
from this event that will enable us to improve the overall reliability of the grid. 

Thank you. 


NERC Testimony 

Subcommittee on Oversight of Government Management, the Federal Workforce, and the 
District of Columbia 

November 20, 2003 Page 5 
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Attachment A 


North American Electric Reliability Council 

Princeton Forrestal Village, 116-390 Village Boulevard, Princeton, New Jersey 08540-5731 


August 14, 2003 Blackout Investigation 
NERC Steering Group 

SCOPE 

August 27, 2003 


Scope 

The NERC Steering Group steers the formulation and implementation of the NERC blackout investigation plan, 
reviews the milestone progress and results, and recommends improvements. The Steering Group provides a 
perspective of industry experts in power system planning, design, and operation. 

Members 

The members of the NERC Steering Group are: 

Paul F. Barber, Facilitator Yakout Mansour 

Barber Energy Senior Vice President 

System Operations & Asset Management 
British Columbia Transmission Corporation 

William (Bill) K. Newman 

Senior Vice President 
Transmission Planning and Operations 
Southern Company Services, Inc. 


Terry M. Winter 

President and Chief Executive Officer 
California ISO 


M. Dale McMaster 

Executive Vice President-Operations and 
Reliability 

Alberta Electric System Operator 


W. Terry Boston 

Executive Vice President 
Transmission/Power Supply Group 
Tennessee Valley Authority 


Mark Fidrych 

Power Operations Specialist 
Western Area Power Administration 


Sam R. Jones 

Chief Operating Officer 

Electric Reliability Council of Texas 


Phone 609-452-8060 - Fax 609-452-9550 - URL www.nerc.com 


GWC August 27, 2003 
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Biographies 


Paul F. Barber, Ph.D, 

Barber Energy 

Dr. Barber provides transmission and engineering services to the electric power industry in areas of governance, 
strategic planning, electric grid management, and power system reliability. He previously served as the Chair of the 
NERC Market Interface Committee and as the Vice Chair (Transmission Customers) of the Northeast Power 
Coordinating Council (NPCC). Dr. Barber joined Boston-based Citizens Power & Light, providing transmission and 
engineering technical expertise and support to all business lines of Citizens Power & Light and its successors. Dr. 
Barber served on the NERC Board of Trustees as well as the Boards of the Mid- Atlantic Area Council, Western 
Systems Coordinating Council (WSCC), and the three Regional Transmission Associations in the Western 
Interconnection. Prior to 1994, Dr. Barber served a 28-year career as an officer in the U.S. Army Corps of Engineers 
rising to the rank of Colonel. Dr. Barber received his BS degree from the U.S. Military Academy and MS degrees in 
electrical engineering and civil engineering from the University of Illinois. He completed a Ph.D. degree in electric 
power engineering from Rensselaer Polytechnic Institute in 1988. He has been registered in the State of Illinois as a 
professional engineer since 1974. 

W. Terry Boston 

Executive Vice President, Transmission/Power Supply Group 
Tennessee Valley Authority 

Terry Boston is Executive Vice President of the Tennessee Valley Authority’s Transmission/Power Supply Group. 
Mr. Boston is the senior officer responsible for the planning, building, operation, and maintenance of TVA’s 
transmission and power supply network. He joined TVA as a power supply engineer in 1972, and was named head 
of the Power Supply Group in 1980. Over the next 16 years, he directed three TVA divisions in succession: 
Transmission, Regional Operations, and Electric System Reliability. Mr. Boston has served for six years on the 
NERC Engineering Committee and Transmission Task Force, and is on the NERC Stakeholders Committee. He is 
vice president of CIGRE, the International Council on Large Electric Systems, and vice president of CERTS (the 
Consortium for Electric Reliability Technology Solutions). Boston holds a B.S. in engineering from Tennessee 
Technological University and an M.S. in engineering administration from the University of Tennessee. 

Mark Fidrych 

Power Operations Specialist 

Western Area Power Administration 

Mark E. Fidrych has served as the Manager of Western Area Power Administration’s Rocky Mountain Desert 
Southwest Reliability Center. Mr. Fidrych began his career with WAPA in 1979, working in maintenance and 
marketing, with the majority of his career having been in power system operations. He directed activities in the 
computer systems and power scheduling divisions before becoming the Operations Manager in 1990. A 1972 
graduate of the University of Rhode Island, Mr. Fidrych received a bachelor's degree in electrical engineering. In 
1980, he received a master's degree in public administration from the University of Colorado. Mr. Fidrych is the 
present Chair of the NERC Operating Committee. He has also served as the Chair of the NERC Security Coordinator 
and the Operating Reliability Subcommittees. 

Sam R. Jones 

Chief Operating Officer 

Electric Reliability Council of Texas 

Sam R. Jones became the first Director of the Electric Reliability Council of Texas (ERCOT) on December I, 1996. 
In March 2000, he was appointed as the Executive Vice President and Chief Operating Officer of ERCOT. Prior to 
joining ERCOT, Mr. Jones was employed by the City of Austin, Texas, Electric Utility for over 35 years. With the 
City of Austin, he held engineering and management positions in the areas of distribution, transmission, substation, 
generation and system operations. He was responsible for the development of Austin’s first energy control center. 

He retired from the City of Austin as Director of Generation and Energy Control. He has been active in inter-utility 
reliability work for over 1 9 years. He is a two-time past chair of the ERCOT Operating Subcommittee, and a current 
Vice-Chair of the NERC Operating Committee, and a past chair (or member) of numerous NERC and ERCOT 
subcommittees and task forces. Mr. Jones has a degree in Electrical Engineering from the University of Texas at 
Austin and is a Registered Professional Engineer in Texas. 
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Yakout Mansour 

Senior Vice President, System Operations & Asset Management 
British Columbia Transmission Corporation 

Yakout Mansour is Senior Vice President of System Operations & Asset Management of the British Columbia 
Transmission Corporation. Previously, he served as the Vice President of the Grid Operations and Inter-Utility 
Affairs division of BC Hydro, responsible for BC Hydro’s transmission, distribution and generation dispatch 
operations as well as the development of policies and practices related to inter-utility transmission access. Mr. 
Mansour currently serves as BC Hydro’s principal representative and board member on the RTO West filing utilities 
structure and has been the Canadian representative in the RTO consultation process. Mr. Mansour is a registered 
Professional Engineer in the Provinces of British Columbia and Alberta with over 30 years experience in power 
system planning, system and market operation, design and research. He is a Fellow of IEEE, has authored and co- 
authored over 100 papers and special publications of IEEE and other international professional institutions, has 
provided training and consulting services around the world, and holds U.S. and Canadian patents. 

Dale McMaster, P.Eng. 

Executive Vice-President, Operations and Reliability 
Alberta Electric System Operator (AESO) 

Dale McMaster is Executive Vice-President, Operations and Reliability for the Alberta Electric System Operator 
(AESO). The AESO integrates the functions of the Power Pool of Alberta, the Transmission Administrator of 
Alberta, and provincial load settlement. Mr. McMastcr’s knowledge of system planning and his overall industry 
experience integrates the AESO’s operational and planning areas. As President and System Controller, Mr. 
McMaster played a key role during the integration of the former Power Pool and the Transmission Administration. 
Mr. McMaster joined the former Power Pool of Alberta in 1996 as Chief Operations Officer, with responsibility for 
the system control function, the ongoing development of the Alberta electric energy market, and strategic planning. 
He is an electrical engineer with more than 25 years of experience in power systems in Canada and abroad. Mr. 
McMaster received his degree in electrical engineering from the University of Saskatchewan and held a variety of 
senior management positions at SaskPower, SNC-Lavalin, and Acres International. He is a member of the 
Association of Professional Engineers, and the Canadian Electricity Association. 

William K. Newman 

Senior Vice President, Transmission Planning & Operations 
Southern Company 

William K. Newman began his career with Georgia Power Company in 1966 and progressed through positions of 
increasing responsibility at Georgia Power for 18 years. In-1984, he assumed the position of General Manager, 
Power Operations, at Mississippi Power Company, was promoted to Director of Power Delivery in 1988 and named 
Vice President, Power Generation and Delivery, in 1989. His responsibilities at Mississippi Power Company 
included the areas of fuels, environmental, generating plants, transmission, and system operations. He transferred to 
Southern Company Services in 1 992 as Vice President, Operating and Planning Services and was named Senior Vice 
President, Transmission Planning and Operations in 1995. He is responsible for planning and operation of the 
Southern electric system's network transmission grid in order to provide economic, reliable service to all users. Mr. 
Newman has served in numerous academic and professional organizations and is currently Chairman, Southeastern 
Electric Reliability Council. He is a Registered Professional Engineer in the states of Georgia and Mississippi. 

Terry M. Winter, 

President and Chief Executive Officer 
California ISO 

Terry M. Winter is President and Chief Executive Officer of the California Independent System Operator (ISO), a 
position he has held since March 1, 1999. Mr. Winter was formerly Chief Operating Officer of the California ISO, 
having accepted the position in August 1997. He assisted in developing operations from the ground up and oversaw 
the integration of the transmission systems of Southern California Edison, Pacific Gas & Electric, and San Diego Gas 
& Electric when the California ISO assumed control of the state’s open market transmission grid on March 31, 1998. 
Mr. Winter was formerly the Division Manager of San Diego Gas & Electric’s power operations. His 21-year career 
with SDG&E focused on power operations, transmission engineering and project management. Prior to his tenure 
with SDG&E he worked on electrical transmission and distribution engineering for Arizona’s Salt River Project for 
10 years. Mr. Winter holds professional engineering licenses in both California and Arizona. Mr. Winter graduated 
from the University of Idaho with a Bachelor of Science degree in Electrical Engineering. 
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Attachment B 

MICHEHL R. GENT 
President and CEO 


North American Electric Reliability Council 

Princeton Forrestal Village, 1 16-390 Village Boulevard, Princeton, New Jersey 08540-5731 


October 15, 2003 


Name 

Address 

Dear Name: 


Near-Term Actions to Assure Reliable Operations 

On October 10, 2003, the NERC Board of Trustees, with the endorsement of its Stakeholders 
Committee, directed that the following letter be sent to the CEOs of all NERC control areas and 
reliability coordinators. 

NERC is assisting the U.S.-Canada Joint Task Force’s investigation of the August 14, 2003, 
blackout that affected parts of the Midwest and Northeast United States, and Ontario, Canada. 
Although considerable progress has been made in the investigation to determine what happened, 
an understanding of the causes of the outage is still being developed through analysis by teams of 
experts. 

The reliability of the North American bulk electric systems, including the avoidance of future 
cascading outages, is of paramount importance to NERC and its stakeholders. Pending the 
outcome of the final report on the outage, NERC emphasizes to all entities responsible for the 
reliable operation of bulk electric systems the importance of assuring those systems are operated 
within their design criteria and within conditions known to be reliable through analytic study. If 
the power system enters an unanalyzed state, system operators must have the authority and the 
capability to take emergency actions to return the power system to a safe condition. 

NERC requests that each entity in North America that operates a control area and each NERC 
reliability coordinator review the following list of reliability practices to ensure their 
organizations are within NERC and regional reliability council standards and established good 
utility practices. NERC further requests that within 60 days, each entity report in writing to their 
respective regional reliability council, with a copy to NERC, that such a review has been 
completed and the status of any necessary corrective actions. This brief list of near-term actions 
is not in any way intended to diminish the need to comply with all NERC and regional reliability 
council standards and good utility practices. 

1. Voltage and Reactive Management: Ensure sufficient voltage support for reliable 
operations. 


Phone 609-452-8060 ■ Fax 609-452-9550 ■ URL www.nerc.com 
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Name 

October 15,2003 

Page Two 

• Establish a daily voltage/reactive management plan, assuring an adequate static and 
dynamic reactive supply under a credible range of system dispatch patterns. 

• During anticipated heavy load days, or conditions of system stress such as caused by 
heavy wide-area transfers, ensure all possible VAR supplies are verified and available, 
and VAR supplies are applied early in the day ahead of load pickup. 

• Reserve sufficient dynamic reactive supply (e.g. online generation and other dynamic 
VAR resources) to meet regional operating criteria and system needs. 

• In accordance with NERC and regional practices maintain voltage schedules of all bulk 
electric transmission facilities above 95% of nominal values and in conformance with 
regional criteria. 

• Report any low voltage limit violations at critical high voltage transmission facilities to 
the reliability coordinator. 

• Ensure all interconnected generators that have, or are required to have, automatic voltage 
regulation (AVR) are operating under AVR. 

• Coordinate potential differences of voltage criteria and schedules between systems and - 
ensure these differences are factored into daily operations. 

2. Reliability Communications: Review, and as necessary strengthen, communication 
protocols between control area operators, reliability coordinators, and ISOs. 

• Share the status of key facilities with other appropriate control area operators, reliability 
coordinators, and ISOs. 

• Control area operators, reliability coordinators, and ISOs should conduct periodic 
conference calls to discuss expected system conditions and notify all neighboring systems 
of any unusual conditions. Conduct additional calls as needed for system critical days. 

3. Failures of System Monitoring and Control Functions: Review and as 
necessary, establish a formal means to immediately notify control room personnel when 
SCADA or EMS functions, that are critical to reliability, have failed and when they are 
restored. 

• Establish an automated method to alert power system operators and technical support 
personnel when power system status indications are not current, or that alarms are not 
being received or annunciated. 

• Determine what backup capabilities can be utilized when primary alarm systems are 
unavailable. If a backup to failed alarms is not immediately available, then monitoring 
and control should be transferred in accordance with approved backup plans. 

• Identify and implement procedures to move to ‘conservative system operations’ when 
operators are unsure about next contingency outcomes (i.e., unstudied conditions, loss of 
SCADA or EMS visibility, unexplained or unknown power system conditions). 

• Ensure all critical computer and communication systems have a backup power supply, 
and the backup supply is periodically tested. 

• Ensure that system operators have a clear understanding of the impact to their energy 
management system control functions whenever their transaction tagging and scheduling 
systems fail. Identify and implement appropriate contingency procedures for loss of real- 
time ACE and AGC control. 
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Name 

October 1 5, 2003 

Page Three 

4. Emergency Action Plans: Ensure that emergency action plans and procedures are in 
place to safeguard the system under emergency conditions by defining actions operators 
may take to arrest disturbances and prevent cascading. 

• Actions might include but should not be limited to acting immediately to reduce 
transmission loading, ordering redispatch, requiring maximum reactive output from 
interconnected resources, and shedding load without first implementing normal operating 
procedures. 

• Ensure operators know, not only that they have the authority to shed load under 
emergencies, but that, in addition, they are expected to exercise that authority to prevent 
cascading. 

5. Training for Emergencies: Ensure that all operating staff are trained and certified, if 
required, and practice emergency drills that include criteria for declaring an emergency, 
prioritized action plans, staffing and responsibilities, and communications. 

6. Vegetation Management: Ensure high voltage transmission line rights of way are free 
of vegetation and other obstructions that could contact an energized conductor within the 
normal and emergency ratings of each line. 


Sincerely, 
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1. Introduction 


On August 14, 2003, large portions of the Midwest 
and Northeast United States and Ontario, Canada, 
experienced an electric power blackout. The out- 
age affected an area with an estimated 50 million 
people and 61,800 megawatts (MW) of electric 
load in the states of Ohio, Michigan, Pennsylva- 
nia, New York, Vermont, Massachusetts, Connect- 
icut, and New Jersey and the Canadian province of 
Ontario. The blackout began a few minutes after 
4:00 pm Eastern Daylight Time (16:00 EDT), and 
power was not restored for 2 days in some parts of 
the United States. Parts of Ontario suffered rolling 
blackouts for more than a week before full power 
was restored. 

On August 15, President George W. Bush and 
Prime Minister Jean Chretien directed that a joint 
U.S.-Canada Power System Outage Task Force be 
established to investigate the causes of the black- 
out and how to reduce the possibility of future 
outages. They named U.S. Secretary of Energy 
Spencer Abraham and Herb Dhaliwal, Minister of 
Natural Resources, Canada, to chair the joint Task 
Force. Three other U.S. representatives and three 
other Canadian representatives were named to the 
Task Force. The U.S. members are Tom Ridge, 
Secretary of Homeland Security; Pat Wood, Chair- 
man of thejrederal Energy Regulatory Commis- 
sion; and Nils Diaz, Chairman of the Nuclear 
Regulatory Commission. The Canadian members 
are Deputy Prime Minister John Manley, Deputy 
Prime Minister; Kenneth Vollman, Chairman of 
the National Energy Board; and Linda J. Keen, 
President and CEO of the Canadian Nuclear Safety 
Commission. 

The Task Force divided its work into two phases: 

♦ Phase 1: Investigate the outage to determine its 
causes and why it was not contained. 

♦ Phase II: Develop recommendations to reduce 
the possibility of future outages and minimize 
the scope of any that occur. 


The Task Force created three Working Groups to 
assist in the Phase I investigation of the blackout — 
an Electric System Working Group (ESWG), a 
Nuclear Working Group (NWG), and a Security 
Working Group (SWG). They were tasked with 
overseeing and reviewing investigations of the 
conditions and events in their respective areas and 
determining whether they may have caused or 
affected the blackout. The Working Groups are 
made up of State and provincial representatives, 
Federal employees, and contractors working for 
the U.S. and Canadian government agencies repre- 
sented on the Task Force. 

This document provides an Interim Report, for- 
warded by the Working Groups, on the findings of 
the Phase I investigation. It presents the facts that 
the bi-national investigation has found regarding 
the causes of the blackout on August 14, 2003. The 
Working Groups and their analytic teams are con- 
fident of the accuracy of these facts and the analy- 
sis built upon them. This report does not offer 
speculations or assumptions not supported by 
evidence and analysis. Further, it does not attempt 
to draw broad conclusions or suggest policy rec- 
ommendations; that task is to be undertaken in 
Phase II and is beyond the scope of the Phase I 
investigation. _ 

This report will now be subject to public review 
and comment. The Working Groups will consider 
public commentary on the Interim Report and will 
oversee and review any additional analyses and 
investigation that may be required. This report 
will be finalized and made a part of the Task Force 
Final Report, which will also contain recommen- 
dations on how to minimize the likelihood and 
scope of future blackouts. 

The Task Force will hold three public forums, or 
consultations, in which the public will have the 
opportunity to comment on this Interim Report 
and to present recommendations for consider- 
ation by the Working Groups and the Task Force. 
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The public may also submit comments and recom- 
mendations to the Task Force electronically or by 
mail. Electronic submissions may be sent to: 

poweroutage@nrcan.gc.ca 

and 

blackout.report@hq.doe.gov. 

Paper submissions may be sent by mail to: 

Dr. Nawal Kamel 

Special Adviser to the Deputy Minister 
Natural Resources Canada 
21st Floor 
580 Booth Street 
Ottawa, ON KlA 0E4 
and 

Mr. James W. Glotfelty 
Director, Office of Electric Transmission 
and Distribution 
U.S. Department of Energy 
1000 Independence Avenue, S.W. 

Washington, DC 20585 

This Interim Report is divided into eight chapters, 
including this introductory chapter: 

♦ Chapter 2 provides an overview of the institu- 
tional framework for maintaining and ensuring 
the reliability of the bulk power system in North 
America, with particular attention to the roles 
and responsibilities of several types of reliabil- 
ity-related organizations. 


♦ Chapter 3 discusses conditions on the regional 
power system before August 14 and on August 
14 before the events directly related to the 
blackout began. 

♦ Chapter 4 addresses the causes of the blackout, 
with particular attention to the evolution of 
conditions on the afternoon of August 14, start- 
ing from normal operating conditions, then 
going into a period of abnormal but still poten- 
tially manageable conditions, and finally into 
an uncontrollable cascading blackout. 

♦ Chapter 5 provides details on the cascade phase 
of the blackout. 

♦ Chapter 6 compares the August 14, 2003, black- 
out with previous major North American power 
outages. 

♦ Chapter 7 examines the performance of the 
nuclear power plants affected by the August 14 
outage. 

♦ Chapter 8 addresses issues related to physical 
and cyber security associated with the outage. 

This report also includes four appendixes: a de- 
scription of the investigative process that pro- 
vided the basis for this report, a list of electricity 
acronyms, a glossary of electricity terms, and three 
transmittal letters pertinent to this report from the 
three Working Groups. 
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2. Overview of the North American Electric Power 
System and Its Reliability Organizations 


The North American Power Grid 
Is One Large, Interconnected 
Machine 

The North American electricity system is one of 
the great engineering achievements of the past 100 
years. This electricity infrastructure represents 
more than $1 trillion in asset value, more than 

200.000 miles (320,000 kilometers) of transmis- 
sion lines operating at 230,000 volts and greater, 

950.000 megawatts of generating capability, and 
nearly 3,500 utility organizations serving well 
over 100 million customers and 283 million 
people. 

Modem society has come to depend on reliable 
electricity as an essential resource for national 
security; health and welfare; communications; 
finance; transportation; food and water supply; 
heating, cooling, and lighting; computers and 
electronics; commercial enterprise; and even 
entertainment and leisure — in short, nearly all 
aspects of modem life. Customers have grown to 
expect that electricity will almost always be avail- 
able when needed at the flick of a switch. Most 
customers have also experienced local outages 
caused by a car hitting a power pole, a construc- 
tion crew accidentally damaging a cable, or a 

Figure 2.1. Basic Structure of the Electric System 


lightning storm. What is not expected is the occur- 
rence of a massive outage on a calm, warm day. 
Widespread electrical outages, such as the one 
that occurred on August 14, 2003, are rare, but 
they can happen if multiple reliability safeguards 
break down. 

Providing reliable electricity is an enormously 
complex technical challenge, even on the most 
routine of days. It involves real-time assessment, 
control and coordination of electricity production 
at thousands of generators, moving electricity 
across an interconnected network of transmission 
lines, and ultimately delivering the electricity to 
millions of customers by means of a distribution 
network. 

As shown in Figure 2.1, electricity is produced at 
lower voltages (10,000 to 25,000 volts) at genera- 
tors from various fuel sources, such as nuclear, 
coal, oil, natural gas, hydro power, geothermal, 
photovoltaic, etc. Some generators are owned by 
the same electric utilities that serve the end-use 
customer; some are owned by independent power 
— producers (IPPs); and others are owned by cus- 
tomers “themselves — particularly large industrial 
customers. 

Electricity from generators is “stepped up” to 
higher voltages for transportation in bulk over 
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transmission lines. Operating the transmission 
lines at high voltage (i.e., 230,000 to 765,000 volts) 
reduces the losses of electricity from conductor 
heating and allows power to be shipped economi- 
cally over long distances. Transmission lines are 
interconnected at switching stations and substa- 
tions to form a network of lines and stations called 
the power “grid.” Electricity flows through the 
interconnected network of transmission lines 
from the generators to the loads in accordance 
with the laws of physics — along “paths of least 
resistance,” in much the same way that water 
flows through a network of canals. When the 
power arrives near a load center, it is “stepped 
down” to lower voltages for distribution to cus- 
tomers. The bulk power system is predominantly 
an alternating current (AC) system, as opposed to 
a direct current (DC) system, because of the ease 
and low cost with which voltages in AC systems 
can be converted from one level to another. Some 
larger industrial and commercial customers take 
service at intermediate voltage levels (12,000 to 
115,000 volts), but most residential customers 
take their electrical service at 120 and 240 volts. 

While the power system in North America is com- 
monly referred to as “the grid," there are actually 
three distinct power grids or “interconnections” 
(Figure 2.2). The Eastern Interconnection includes 
the eastern two-thirds of the continental United 
States and Canada from Saskatchewan east to the 
Maritime Provinces. The Western Interconnection 
includes the western third of the continental 
United States (excluding Alaska), the Canadian 
Provinces of Alberta and British Columbia, and a 
portion of Baja California Norte, Mexico. The third 
interconnection comprises roost of the state of 


Figure 2.2. NERC Interconnections 



Texas. The three interconnections are electrically 
independent from each other except for a few 
small direct current (DC) ties that link them. 
Within each interconnection, electricity is pro- 
duced the instant it is used, and flows over virtu- 
ally all transmission lines from generators to 
loads. 

The northeastern portion of the Eastern Intercon- 
nection (about 10 percent of the interconnection’s 
total load) was affected by the August 14 blackout. 
The other two interconnections were not 
affected. 1 

Planning and Reliable Operation 
of the Power Grid Are Technically 
Demanding 

Reliable operation of the power grid is complex 
and demanding for two fundamental reasons: 

♦ First, electricity flows at the speed of light 
(186,000 miles per second or 297,600 kilome- 
ters per second) and is not economically 
storable in large quantities. Therefore electric- 
ity must be produced the instant it is used. 

♦ Second, the flow of alternating current (AC) 
electricity cannot be controlled like a liquid or 
gas by opening or closing a valve in a pipe, or 
switched like calls over a long-distance tele- 
phone network. Electricity flows freely along all 
available paths from the generators to the loads 
in accordance with the laws of physics — divid- 
ing among all connected flow paths in the net- 
work, in inverse proportion to the^ impedance 
(resistance plus reactance) on each path. 

Maintaining reliability is a complex enterprise 
that requires trained and skilled operators, sophis- 
ticated computers and communications, and care- 
ful planning and design. The North American 
Electric Reliability Council (NERC) and its ten 
Regional Reliability Councils have developed sys- 
tem operating and planning standards for ensur- 
ing the reliability of a transmission grid that are 
based on seven key concepts: 

♦ Balance power generation and demand 
continuously. 

♦ Balance reactive power supply and demand to 
maintain scheduled voltages. 

♦ Monitor flows over transmission lines and other 
facilities to ensure that thermal (heating) limits 
are not exceeded. 
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♦ Keep the system in a stable condition. 

♦ Operate the system so that it remains in a reli- 
able condition even if a contingency occurs, 
such as the loss of a key generator or transmis- 
sion facility (the “N-l criterion”). 

♦ Plan, design, and maintain the system to oper- 
ate reliably. 

♦ Prepare for emergencies. 

These seven concepts are explained in more detail 

below. 

1 . Balance power generation and demand contin- 
uously. To enable customers to use as much 
electricity as they wish at any moment, produc- 
tion by tire generators must be scheduled or 
“dispatched” to meet constantly changing 
demands, typically on an hourly basis, and then 
fine-tuned throughout the hour, sometimes 
through the use of automatic generation con- 
trols to continuously match generation to actual 
demand. Demand is somewhat predictable, 
appearing as a daily demand curve — in the 
summer, highest during the afternoon and eve- 
ning and lowest in the middle of the night, and 
higher on weekdays when most businesses are 
open (Figure 2.3). 

Failure to match generation to demand causes 
the frequency of an AC power system (nomi- 
nally 60 cycles per second or 60 Hertz) to 
increase (when generation exceeds demand) or 
decrease (when generation is less than demand) 
(Figure 2.4). Random, small variations in fre- 
quency are normal, as loads come on and off 
and generators modify their output to follow the 
demand changes. However, large deviations in 
frequency can cause the rotational speed of gen- 
erators to fluctuate, leading to vibrations that 
can damage generator turbine blades and other 
equipment. Extreme low frequencies can trigger 


Figure 2.3. PJM Load Curve, August 18-24, 2003 



automatic under-frequency “load shedding,” 
which takes blocks of customers off-line in 
order to prevent a total collapse of the electric 
system. As will be seen later in this report, such 
an imbalance of generation and demand can 
also occur when the system responds to major 
disturbances by breaking into separate 
“islands"; any such island may have an excess 
or a shortage of generation, compared to 
demand within the island. 

2. Balance reactive power supply and demand to 
maintain scheduled voltages. Reactive power 
sources, such as capacitor banks and genera- 
tors, must be adjusted during the day to main- 
tain voltages within a secure range pertaining to 
all system electrical equipment (stations, trans- 
mission lines, and customer equipment). Most 
generators have automatic voltage regulators 
that cause the reactive power output of genera- 
tors to increase or decrease to control voltages to 
scheduled levels. Low voltage can cause electric 
system instability or collapse and, at distribu- 
tion voltages, can cause damage to motors and 
the failure of electronic equipment. High volt- 
ages can exceed the insulation capabilities of 
equipment and cause dangerous electric arcs 
(“flashovers”). 

3. Monitor flows over transmission lines and 
other facilities to ensure that thermal (heating) 
limits are not exceeded. The dynamic interac- 
tions between generators and loads, combined 
with the fact that electricity flows freely across 
all interconnected circuits, mean that power 
flow is ever-changing on transmissiommd dis- 
tribution lines. All lines, transformers, and 
other equipment carrying electricity are heated 
by the flow of electricity through them. The 


Figure 2.4. Normal and Abnormal Frequency 
Ranges 
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flow must be limited to avoid overheating and 
damaging the equipment. In the case of over- 
head power lines, heating also causes the metal 
conductor to stretch or expand and sag closer to 
ground level. Conductor heating is also affected 
by ambient temperature, wind, and other fac- 
tors. Flow on overhead lines must be limited to 
ensure that the line does not sag into obstruc- 
tions below such as trees or telephone lines, or 
violate the minimum safety clearances between 
the energized lines and other objects. (A short 
circuit or “flashover” — which can start fires or 
damage equipment — can occur if an energized 
line gels too close to another object). All electric 
lines, transformers and other current-carrying 
devices are monitored continuously to ensure 
that they do not become-overloaded or violate 
other operating constraints. Multiple ratings are 
typically used, one for normal conditions and a 
higher rating for emergencies. The primary 
means of limiting the flow of power on trans- 
mission lines is to adjust selectively the output 
of generators. 

4. Keep the system in a stable condition. Because 
the electric system is interconnected and 
dynamic, electrical stability limits must be 
observed. Stability problems can develop very 
quickly — in just a few cycles (a cycle is l/60th of 
a second) — or more slowly, over seconds or 
minutes. The main concern is to ensure that 
generation dispatch and the resulting power 
flows and voltages are such that the system is 
stable at all times. (As will be described later in 
this report, part of the Eastern Interconnection 
became unstable on August 14, resulting in a 
cascading outage over a wide area.) Stability 


limits, like thermal limits, are expressed as a 
maximum amount of electricity that can be 
safely transferred over transmission lines. 
There are two types of stability limits: (1) Volt- 
age stability limits are set to ensure that the 
unplanned loss of a line or generator (which 
may have been providing locally critical reac- 
tive power support, as described previously) 
will not cause voltages to fall to dangerously 
low levels. If voltage falls too low, it begins to 
collapse uncontrollably, at which point auto- 
matic relays either shed load or trip generators 
to avoid damage. (2) Power (angle) stability lim- 
its are set to ensure that a short circuit or an 
unplanned loss of a line, transformer, or genera- 
tor will not cause the remaining generators and 
loads being served to lose synchronism with 
one another. (Recall that all generators and 
loads within an interconnection must operate at 
or very near a common 60 Hz frequency.) Loss 
of synchronism with the common frequency 
means generators are operating out-of-step with 
one another. Even modest losses of synchro- 
nism can result in damage to generation equip- 
ment. Under extreme losses of synchronism, 
the grid may break apart into separate electrical 
islands; each island would begin to maintain its 
own frequency, determined by the load/genera- 
tion balance within the island. 

5. Operate the system so that it remains in a reli- 
able condition even if a contingency occurs, 
such as the loss of a key generator or transmis- 
sion facility (the “N minus 1 criterion”). The 
central organizing principle of electricity reli- 
ability management is to plan for the unex- 
pected. The unique features of electricity mean 
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that problems, when they arise, can spread and 
escalate very quickly if proper safeguards are 
not in place. Accordingly, through years of 
experience, the industry has developed a 
sequence of defensive strategies for maintaining 
reliability based on the assumption that equip- 
ment can and will fail unexpectedly upon 
occasion. 

This principle is expressed by the requirement 
that the system must be operated at all times to 
ensure that it will remain in a secure condition 
(generally within emergency ratings for current 
and voltage and within established stability 
limits) following the loss of the most important 
generator or transmission facility (a “worst sin- 
gle contingency”). This is called the “N-l crite- 
rion.” In other words, because a generator or 
line trip can occur at any time from random fail- 
ure, the power system must be operated in a 
preventive mode so that the loss of the most 
important generator or transmission facility 
does not jeopardize the remaining facilities in 
the system by causing them to exceed their 
emergency ratings or stability limits, which 
could lead to a cascading outage. 

Further, when a contingency does occur, the 
operators are required to identify and assess 
immediately the new worst contingencies, 
given the changed conditions, and promptly 
make any adjustments needed to ensure that if 
one of them were to occur, the system would 
still remain operational and safe. NERC operat- 
ing policy requires that the system be restored 
as soon as practical but within no more than 30 
~ minutes to compliance with normal limits, and 
to a condition where it can once again with- 
stand the next-worst single contingency with- 
out violating thermal, voltage, or stability 
limits. A few areas of the grid are operated to 
withstand the concurrent loss of two or more 
facilities (i.e., "N-2"). This may be done, for 
example, as an added safety measure to protect 
a densely populated metropolitan area or when 
lines share a common structure and could be 
affected by a common failure mode, e.g., a sin- 
gle lightning strike. 

6. Plan, design, and maintain the system to oper- 
ate reliably. Reliable power system operation 
requires far more than monitoring and control- 
ling the system in real-time. Thorough plan- 
ning, design, maintenance, and analysis are 
required to ensure that the system can be oper- 
ated reliably and within safe limits, Short-term 


planning addresses day-ahead and week-ahead 
operations planning; long-term planning 
focuses on providing adequate generation 
resources and transmission capacity to ensure 
that in the future the system will be able to 
withstand severe contingencies without experi- 
encing widespread, uncontrolled cascading 
outages. 

A utility that serves retail customers must esti- 
mate future loads and, in some cases, arrange 
for adequate sources of supplies and plan ade- 
quate transmission and distribution infrastruc- 
ture. NERC planning standards identify a range 
of possible contingencies and set corresponding 
expectations for system performance under sev- 
eral categories of possible events. Three catego- 
ries represent the more probable types of events 
that the system must be planned to withstand. 
A fourth category represents “extreme events” 
that may involve substantial loss of customer 
load and generation in a widespread area. NERC 
planning standards also address requirements 
for voltage support and reactive power, distur- 
bance monitoring, facility ratings, system mod- 
eling and data requirements, system protection 
and control, and system restoration. 

7. Prepare for emergencies. System operators are 
required to take the steps described above to 
plan and operate a reliable power system, but 
emergencies can still occur because of external 
factors such as severe weather, operator error, 
or equipment failures that exceed planning, 
design, or operating criteria. For these rare 
events, the operating entity is required to have 
“emergency procedures covering a credible 
range of emergency scenarios. Operators must 
be trained to recognize and take effective action 
in response to these emergencies. To deal with a 
system emergency that results in a blackout, 
such as the one that occurred on August 14, 
2003, there must be procedures and capabilities 
to use “black start” generators (capable of 
restarting with no external power source) and to 
coordinate operations in order to restore the 
system as quickly as possible to a normal and 
reliable condition. 

Reliability Organizations Oversee 
Grid Reliability in North America 

NERC is a non-governmental entity whose mis- 
sion is to ensure that the bulk electric system in 

North America is reliable, adequate and secure. 
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The organization was established in 1968, as a 
result of the Northeast blackout in 1965. Since its 
inception, NERC has operated as a voluntary orga- 
nization, relying on reciprocity, peer pressure and 
the mutual self-interest of all those involved to 
ensure compliance with reliability requirements. 
An independent board governs NERC. 

To fulfill its mission, NERC: 

♦ Sets standards for the reliable operation and 
planning of the bulk electric system. 

♦ Monitors and assesses compliance with stan- 
dards for bulk electric system reliability. 

♦ Provides education and training resources to 
promote bulk electric system reliability. 

♦ Assesses, analyzes and reports on bulk electric 
system adequacy and performance. 

♦ Coordinates with Regional Reliability Councils 
and other organizations. 

♦ Coordinates the provision of applications 
(tools), data and services necessary to support 
the reliable operation and planning of the bulk 
electric system. 

♦ Certifies reliability service organizations and 
personnel. 

♦ Coordinates critical infrastructure protection of 
the bulk electric system. 

♦ Enables the reliable operation of the intercon- 
nected bulk electric system by facilitating infor- 
mation exchange and coordination among 
reliability service organizations. 


Figure 2.5. NERC Regions 



Recent changes in the electricity industry have 
altered many of the traditional mechanisms, 
incentives and responsibilities of the entities 
involved in ensuring reliability, to the point that 
the voluntary system of compliance with reliabil- 
ity standards is generally recognized as not ade- 
quate to current needs. 2 NERC and many other 
electricity organizations support the development 
of a new mandatory system of reliability standards 
and compliance, backstopped in the United States 
by the Federal Energy Regulatory Commission. 
This will require federal legislation in the United 
States to provide for the creation of a new electric 
reliability organization with the statutory author- 
ity to enforce compliance with reliability stan- 
dards among all market participants. Appropriate 
government entities in Canada and Mexico are 
prepared to take similar action, and some have 
already done so. In the meantime, NERC encour- 
ages compliance with its reliability standards 
through an agreement with its members. 

NERC’s members are ten Regional Reliability 
Councils. (See Figure 2.5 for a map showing the 
locations and boundaries of the regional councils.) 
The regional councils and NERC have opened 
their membership to include all segments of the 
electric industry: investor-owned utilities; federal 
power agencies; rural electric cooperatives; state, 
municipal and provincial utilities; independent 
power producers; power marketers; and end-use 
customers. Collectively, the members of the NERC 
regions account for virtually all the electricity sup- 
plied in the United States, Canada, and a portion 
of Baja California Norte, Mexico. The ten regional 
councils jointly fund NERC and adapt NERC slam 
dards to meet the needs of their regions. The 
August 14 blackout affected three NERC regional 
reliability councils — East Central Area Reliability 
Coordination Agreement (ECAR), Mid-Atlantic 
Area Council (MAAC), and Northeast Power Coor- 
dinating Council (NPCC), 

“Control areas” are the primary operational enti- 
ties that are subject to NERC and regional council 
standards for reliability. A control area is a geo- 
graphic area within which a single entity, Inde- 
pendent System Operator (ISO), or Regional 
Transmission Organization (RTO) balances gener- 
ation and loads in real time to maintain reliable 
operation. Control areas are linked with each 
other through transmission interconnection tie 
lines. Control area operators control generation 
directly to maintain their electricity interchange 
schedules with other control areas. They also 
operate collectively to support the reliability of 
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their interconnection. As shown in Figure 2.6, 
there are approximately 140 control areas in North 
America. The control area dispatch centers have 
sophisticated monitoring and control systems and 
are staffed 24 hours per day, 365 days per year. 

Traditionally, control areas were defined by utility 
service area boundaries and operations were 
largely managed by vertically integrated utilities 
that owned and operated generation, transmis- 
sion, and distribution. While that is still true in 
some areas, there has been significant restructur- 
ing of operating functions and some consolidation 
of control areas into regional operating entities. 
Utility industry restructuring has led to an 
unbundling of generation, transmission and dis- 
tribution activities such that the ownership and 
operation of these assets have been separated 
either functionally or through the formation of 
independent entities called Independent System 
Operators (ISOs) and Regional Transmission 
Organizations (RTOs). 


♦ ISOs and RTOs in the United States have been 
authorized by FERC to implement aspects of the 
Energy Policy Act of 1992 and subsequent FERC 
policy directives. 

♦ The primary functions of ISOs and RTOs are to 
manage in real time and on a day-ahead basis 
the reliability of the bulk power system and the 
operation of wholesale electricity markets 
within their footprint. 

♦ ISOs and RTOs do not own transmission assets; 
they operate or direct the operation of assets 
owned by their members. 

♦ ISOs and RTOs may be control areas them- 
selves, or they may encompass more than one 
control area. 

♦ ISOs and RTOs may also be NERG Reliability 
Coordinators, as described below. 

Five RTOs/ISOs are within the area directly 

affected by the August 14 blackout. They are: 


Figure 2.6. NERC Regions and Control Areas 
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♦ Midwest Independent System Operator (MISO) 

♦ PJM Interconnection (PJM) 

♦ New York Independent System Operator 
(NYISO) 

♦ New England Independent System Operator 
(ISO-NE) 

♦ Ontario Independent Market Operator (IMO) 

Reliability coordinators provide reliability over- 
sight over a wide region. They prepare reliability 
assessments, provide a wide-area view of reliabil- 
ity, and coordinate emergency operations in real 
time for one or more control areas. They do not 
participate in the wholesale or retail market func- 
tions. There are currently 18 reliability coordina- 
tors in North America. Figure 2.7 shows the 
locations and boundaries of their respective areas. 

Key Parties in the Pre-Cascade 
Phase of the August 14 Blackout 

The initiating events of the blackout involved two 
control areas — FirstEnergy (FE) and American 
Electric Power (AEPJ — and their respective reli- 
ability coordinators, MISO and PJM (see Figures 
2.7 and 2.8). These organizations and their reli- 
ability responsibilities are described briefly in this 
final subsection. 

1 . FirstEnergy operates a control area in north- 
ern Ohio. FirstEnergy (FE) consists of seven 
electric utility operating companies. Four of 
these companies, Ohio Edison, Toledo Edison, 
The Illuminating' Company, and Penn Power, 
operate in the NERC ECAR region, with MISO 


Figure 2.7. NERC Reliability Coordinators 



serving as their reliability coordinator. These 
four companies now operate as one integrated 
control area managed by FE. 3 

2. American Electric Power (AEP) operates a con- 
trol area in Ohio just south of FE. AEP is both a 
transmission operator and a control area 
operator. 

3. Midwest Independent System Operator 
(MISO) is the reliability coordinator for 
FirstEnergy. The Midwest Independent System 
Operator (MISO) is the reliability coordinator 
for a region of more than one million square 
miles, stretching from Manitoba, Canada in the 
north to Kentucky in the south, from Montana 
in the west to western Pennsylvania in the east. 
Reliability coordination is provided by two 
offices, one in Minnesota, and the other at the 
MISO headquarters in Indiana. Overall, MISO 
provides reliability coordination for 37 control 
areas, most of which are members of MISO. 

4. PJM is AEP’s reliability coordinator. PJM is one 
of the original ISOs formed after FERC orders 
888 and 889, but was established as a regional 
power pool in 1935. PJM recently expanded its 
footprint to include control areas and transmis- 
sion operators within MAIN and ECAR (PJM- 
West). It performs its duties as a reliability coor- 
dinator in different ways, depending on the 
control areas involved. For PJM-East, it is 
both the control area and reliability coordinator 
for ten utilities, whose transmission systems 
span the Mid -Atlantic region of New Jersey, 
most of Pennsylvania, Delaware, Maryland, 
West Virginia, Ohio, Virginia, and the District of 
Columbia. The PJM-West facility has the reli- 
ability coordinator desk for five control areas 
(AEP, Commonwealth Edison, Duquesne Light, 


Figure 2.8. Reliability Coordinators and Control 
Areas in Ohio and Surrounding States 
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Dayton Power and Light, and Ohio Valley Elec- 
tric Cooperative) and three generation-only 
control areas (Duke Energy’s Washington 
County (Ohio) facility, Duke’s Lawrence 
County/Hanging Rock (Ohio) facility, and Alle- 
gheny Energy’s Buchanan (West Virginia) 
facility. 

Reliability Responsibilities of Control 
Area Operators and Reliability 
Coordinators 

1. Control area operators have primary responsi- 
bility for reliability. Their most important 
responsibilities, in the context of this report, 
are: 

N-l criterion. NERC Operating Policy 2.A— 
Transmission Operations: 


“All CONTROL AREAS shall operate so that 
instability, uncontrolled separation, or cas- 
cading outages will not occur as a result of 
the most severe single contingency.” 

Emergency preparedness and emergency 
response. NERC Operating Policy 5 — Emer- 
gency Operations, General Criteria: 

“Each system and CONTROL AREA, shall 
promptly take appropriate action to relieve 
any abnormal conditions, which jeopardize 
reliable Interconnection operation.” 

"Each system, CONTROL AREA, and Region 
shall establish a program of manual and auto- 
matic load shedding which is designed to 
arrest frequency or voltage decays that could 
result in an uncontrolled failure of compo- 
nents of the interconnection." 
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NERC Operating Policy 5. A — Coordination 
with Other Systems: 

“A system, CONTROL AREA, or pool that is 
experiencing or anticipating an operating 
emergency shall communicate its current 
and future status to neighboring systems, 
CONTROL AREAS, or pools and throughout the 
interconnection.... A system shall inform 
other systems ... whenever ... the system’s 
condition is burdening other systems or 
reducing the reliability of the Interconnec- 
tion .... [or whenever] the system’s line load- 
ings and voltage/reactive levels are such that 
a single contingency could threaten the reli- 
ability of the Interconnection.” 

NERC Operating Policy 5.C — Transmission 
System Relief: 

“Action to correct an OPERATING SECURITY 
LIMIT violation shall not impose unaccept- 
able stress on internal generation or transmis- 
sion equipment, reduce system reliability 
beyond acceptable limits, or unduly impose 
voltage or reactive burdens on neighboring 
systems. If all other means fail, corrective 
action may require load reduction.” 

Operating personnel and training: NERC Oper- 
ating Policy 8.B— Training: 

“Each OPERATING AUTHORITY should period- 
ically practice simulated emergencies. The 
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scenarios included in practice situations 
should represent a variety of operating condi- 
tions and emergencies.” 

2, Reliability Coordinators such as MISO and 
PJM are expected to comply with all aspects of 
NERC Operating Policies, especially Policy 9, 
Reliability Coordinator Procedures, and its 
appendices. Key requirements include: 

NERC Operating Policy 9, Criteria for Reliabil- 
ity Coordinators, 5.2: 

Have “detailed monitoring capability of the 
RELIABILITY area and sufficient monitoring 
capability of the surrounding RELIABILITY 
AREAS to ensure potential security violations 
are identified.” 

NERC Operating Policy 9, Functions of Reliabil- 
ity Coordinators, 1.7: 

“Monitor the parameters that may have sig- 
nificant impacts within the RELIABILITY AREA 
and with neighboring RELIABILITY AREAS 
with respect to ... sharing with other 
RELIABILITY COORDINATORS any information 
regarding potential, expected, or actual criti- 
cal operating conditions that could nega- 
tively impact other RELIABILITY AREAS. The 
Reliability Coordinator will coordinate 
with other RELIABILITY COORDINATORS and 
CONTROL AREAS as needed to develop appro- 
priate plans to mitigate negative impacts of 
potential, expected, or actual critical operat- 
ing conditions....” 

NERC Operating Policy 9, Functions of Reliabil- 
ity Coordinators, 6: 

“Conduct security assessment and monitor- 
ing programs to assess contingency situa- 
tions. Assessments shall be made in real time 
and for the operations planning horizon at 
the CONTROL AREA level with any identified 
problems reported to the RELIABILITY CO- 
ORDINATOR. The RELIABILITY COORDINATOR 
is to ensure that CONTROL AREA, RELIABILITY 
AREA, and regional boundaries are suffi- 
ciently modeled to capture any problems 
crossing such boundaries." 

Endnotes 

*The province of Quebec, although considered a part of the 
Eastern Interconnection, is connected to the rest of the East- 
ern Interconnection primarily by DC ties. In this instance, the 
DC ties acted as buffers between portions of the Eastern Inter- 
connection; transient disturbances propagate through them 
less readily. Therefore, the electricity system in Quebec was 
not affected by the outage, except for a small portion of the 
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province’s load that is directly connected to Ontario by AC 
transmission lines. (Although DC ties can act as a buffer 
between systems, the tradeoff is that they do not allow instan- 
taneous generation support following the unanticipated loss 
of a generating unit.) 

2 See, for example. Maintaining Reliability in a Competitive 
Electric Industry (1998), a report to the U.S. Secretary of 
Energy by the Task Force on Electric Systems Reliability; 
National Energy Policy (2001), a report to the President of the 


United States by the National Energy Policy Development 
Group, p. 7-6; and National Transmission Grid Study (2002), 
U.S. Dept, of Energy, pp. 46-48. 

3 The remaining three FE companies, Penelec, Met-Ed, and 
Jersey Central Power & Light, are in the NERC MAAC region 
and have PJM as their reliability coordinator. The focus of this 
report is on the portion of FE in ECAR reliability region and 
within the M1SO reliability coordinator footprint. 
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3. Status of the Northeastern Power Grid 
Before the Blackout Sequence Began 


Summary 

This chapter reviews the state of the northeast por- 
tion of the Eastern Interconnection during the 
days prior to August 14, 2003 and up to 15:05 EDT 
on August 14 to determine whether conditions at 
that time were in some way unusual and might 
have contributed to the initiation of the blackout. 
The Task Force's investigators found that at 15:05 
EDT, immediately before the tripping (automatic 
shutdown) of FirstEnergy’s (FE) Harding-Cham- 
beriin 345-kV transmission line, the system was 
able to be operated reliably following the occur- 
rence of any of more than 800 contingencies, 
including the loss of the Harding-Chamberlin line. 
At that point the system was being operated near 
(but still within) prescribed limits and in compli- 
ance with NERC’s operating policies. 

Determining that the system was in a reliable 
operational state at that time is extremely signifi- 
cant for understanding the causes of the blackout. 
It means that none of the electrical conditions on 
the system before 15:05 EDT was a direct cause of 
the blackout. This eliminates a number of possible 
causes of the blackout, whether individually or in 
combination with one another, such as: 

♦ High power flows to Canada 

♦ System frequency variations 

♦ Low voltages earlier in the day or on prior days 

♦ Low reactive power output from IPPs 

♦ Unavailability of individual generators or trans- 
mission lines. 

It is important to emphasize that establishing 
whether conditions were normal or unusual prior 
to and on August 14 has no direct bearing on the 
responsibilities and actions expected of the orga- 
nizations and operators who are charged with 
ensuring power system reliability. As described in 
Chapter 2, the electricity industry has developed 
and codified a set of mutually reinforcing reliabil- 
ity standards and practices to ensure that system 


operators are prepared for the unexpected. The 
basic assumption underlying these standards and 
practices is that power system elements will fail 
or become unavailable in unpredictable ways. 
Sound reliability management is designed to 
ensure that safe operation of the system will con- 
tinue following the unexpected loss of any key 
element (such as a major generator or key trans- 
mission facility). These practices have been 
designed to maintain a functional and reliable 
grid, regardless of whether actual operating 
conditions are normal. It is a basic principle of 
reliability management that “operators must oper- 
ate the system they have in front of them” — 
unconditionally. 

In terms of day-ahead planning, this means evalu- 
ating and if necessary adjusting the planned 
generation pattern (scheduled electricity transac- 
tions) to change the transmission flows, so that if a 
key facility were lost, the operators would still be 
able to readjust the remaining system and operate 
within safe limits. In terms of real-time operations, 
this means that the system should be operated at 
all times so as to be able to withstandlhe loss of 
any single facility and "Still remain within the sys- 
tem's thermal, voltage, and stability limits. If a 
facility is lost unexpectedly, the system operators 
must determine whether to make operational 
changes to ensure that the remaining system is 
able to withstand the loss of yet another key ele- 
ment and still remain able to operate within safe 
limits. This includes adjusting generator outputs, 
curtailing electricity transactions, and if neces- 
saiy, shedding interruptible and firm customer 
load — i.e., cutting some customers off tempo- 
rarily, and in the right locations, to reduce elec- 
tricity demand to a level that matches what the 
system is then able to deliver safely. 

Electric Demands on August 14 

Temperatures on August 14 were above normal 
throughout the northeast region of the United 
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Slates and in eastern Canada. As a result, electric- 
ity demands were high due to high air condition- 
ing loads typical of warm days in August, though 
not unusually so. System operators had success- 
fully managed higher demands both earlier in the 
summer and in previous years. Recorded peak 
electric demands throughout the region on August 
14 were below peak demands recorded earlier in 
the summer of 2003 (Figure 3.1). 


Figure 3.1. August 2003 Temperatures in the U.S. 
Northeast and Eastern Canada 



Power Flow Patterns 

On August 14, the flow of power through the 
ECAR region was heavy as a result of large trans- 
fers of power from the south (Tennessee, Ken- 
tucky, Missouri, etc.) and west (Wisconsin, 
Minnesota, Illinois, etc.) to the north (Ohio, Mich- 
igan, and Ontario) and east (New York). The desti- 
nations for much of the power were northern 
Ohio, Michigan, PJM, and Ontario (Figure 3.2). 

While heavy, these transfers were not beyond pre- 
vious levels or in directions not seen before 
(Figure 3.3). The level of imports into Ontario on 
August 14 was high but not unusual, and well 
within IMO's import capability. Ontario’s IMO is a 
frequent importer of power, depending on the 
availability and price of generation within 
Ontario. IMO had imported similar and higher 
amounts of power several times during the sum- 
mers of 2002 and 2003. 

System Frequency 

Although system frequency on the Eastern Inter- 
connection was somewhat more variable on 



Load 23,21 5 MW 
Generational, 986 MW 


1341, MW) 


; : Load 27.672 MW 
j Generation 27,376 MW 




Load 19.155 MW 
Generation 18,221 MW 


J8TMW) 


Loadl 2,080 MW 
5ene ration 9,363 MW 


Northern 
1 Indiana 
Public 
Service 


1 Load 27,450 MW 
Generation 28,225 MW 

Y/toerican ^ j 
/■) Ei o e !ih : Pc ive (;■ 
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August 14 prior to 15:05 EDT compared with 
recent history, it was well within the bounds of 
safe operating practices as outlined in NERC oper- 
ating policies. As a result, system frequency varia- 
tion was not a cause of the initiation of the 
blackout. But once the cascade was initiated, the 
large frequency swings that were induced became 



Figure 3.3. Northeast Central Area Scheduled 
Imports and Exports: Summer 2003 Compared to 
August 14, 2003 



Note: Area covered includes ECAR, PJM, Ontario, and New 
York, without imports from the Maritime Provinces, ISO-New 
England, or Hydro-Quebec. 


a principal means by which the blackout spread 
across a wide area (Figure 3.4). 

Assuming stable conditions, the system frequency 
is the same across an interconnected grid at any 
particular moment. System frequency will vary 
from moment to moment, however, depending on 
the second-to-second balance between aggregate 
generation and aggregate demand across the inter- 
connection. System frequency is monitored on a 
continuous basis. 

Generation Facilities Unavailable 
on August 14 

Several key generators in the region were out of 
service going into the day of August 14. On any 
given day, some generation and transmission 
capacity is unavailable; some facilities are out for 
routine maintenance, and others have been forced 
out by an unanticipated breakdown and require 
repairs. August 14, 2003, was no exception (Table 
3.1). 

The generating units that were not available on 
August 14 provide real and reactive power directly 
to the Cleveland, Toledo, and Detroit areas. Under 
standard practice, system operators take into 
account the unavailability of such units and any 


Figure 3.4. Frequency on August 14, 2003, 
up to 15:31 EDT 
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Table 3.1. Generators Not Available on August 14 


Generator ! 

Rating 

Reason 

Davis-Besse Nuclear Unit 

750 MW 

Prolonged NRC-ordered outage beginning on 3/22/02 

Easllake Unit 4 

238 MW 

Forced outage on 8/13/03 

Monroe Unit 1 

817 MW 

Planned outage, taken out of service on 8/8/03 

Cook Nuclear Unit 2 

1 .060 MW 

Outage began on 8/13/03 
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transmission facilities known to be out of service 
in the day-ahead planning studies they perform to 
determine the condition of the system for the next 
day. Knowing the status of key facilities also helps 
operators determine in advance the safe electricity 
transfer levels for the coming day. 

MISO’s day-ahead planning studies for August 14 
took these generator outages and known transmis- 
sion outages into account and determined that the 
regional system could still be operated safely. The 
unavailability of these generation units and trans- 
mission facilities did not cause the blackout. 

Voltages 

During the days before August 14 and throughout 
the morning and mid-day on August 14, voltages 
were depressed in a variety of locations in north- 
ern Ohio because of high air conditioning demand 
and other loads, and power transfers into and 
across the region. (Unlike frequency, which is 
constant across the interconnection, voltage varies 
by location, and operators monitor voltages con- 
tinuously at key locations across their systems.) 
However, actual measured voltage levels at key 
points on FE’s transmission system on the morn- 
ing of August 14 and up to 15:05 EDT were within 
the range previously specified by FE as acceptable. 
Note, however, that many control areas in the 
Eastern Interconnection have set their acceptable 
voltage bands at levels higher than that used 


by FE. For example, AEP’s minimum acceptable 
voltage level is 95% of a line’s nominal rating, as 
compared to FE’s 92%. 1 

Voltage management is especially challenging on 
hot summer days because of high air conditioning 
requirements, other electricity demand, and high 
transfers of power for economic reasons, all of 
which increase the need for reactive power. Oper- 
ators address these challenges through long-term 
planning, day-ahead planning, and real-time 
adjustments to operating equipment. On August 
14, for example, PJM implemented routine voltage 
management procedures developed for heavy load 
conditions. FE also began preparations early in the 
afternoon of August 14, requesting capacitors to 
be restored to service 2 and additional voltage sup- 
port from generators. 3 Such actions were typical 
of many system operators that day as well as on 
other days with high electric demand. As the day 
progressed, operators across the region took addi- 
tional actions, such as increasing plants’ reactive 
power output, plant redispatch, transformer tap 
changes, and increased use of capacitors to 
respond to changing voltage conditions. 

The power flow data for northern Ohio on August 
14 just before the Harding-Chamberlin line trip- 
ped at 15:05 EDT (Figure 3.2) show that FE’s load 
was approximately 12,080 MW. FE was importing 
about 2,575 MW, 21% of its total system needs, 
and generating the remainder. With this high level 
of imports and high air conditioning loads in the 
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metropolitan areas around the southern end of 
Lake Erie, FE’s system reactive power needs rose 
further. Investigation team modeling indicates 
that at 15:00 EDT, with Eastlake 5 out of service, 
FE was a net importer of about 132 MVAr. A 
significant amount of power also was flowing 
through northern Ohio on its way to Michigan and 
Ontario (Figure 3.2). The net effect of this flow pat- 
tern and load composition was to depress voltages 
in northern Ohio. 

Unanticipated Outages of 
Transmission and Generation 
on August 14 

Three significant unplanned outages occurred in 
the Ohio area on August 14 prior to 15:05 EDT. 
Around noon, several Cinergy transmission lines 
in south-central Indiana tripped; at 13:31 EDT, 
FE’s Eastlake 5 generating unit along the south- 
western shore of Lake Erie tripped; at 14:02 EDT, a 
Dayton Power and Light (DPL) line, the Stuart- 
Atlanta 345-kV line in southern Ohio, tripped. 

♦ Transmission lines on the Cinergy 345-, 230-, 
and 138-kV systems experienced a series of out- 
ages starting at 12:08 EDT and remained out of 
service during the entire blackout. The loss of 
these lines caused significant voltage and 
loading problems in the Cinergy area. Cinergy 
made generation changes, and MISO operators 
responded by implementing transmission load 


relief (TLR) procedures to control flows on the 
transmission system in south-central Indiana. 
System modeling by the investigation team (see 
details below, page 20) showed that the loss of 
these lines was not electrically related to subse- 
quent events in northern Ohio that led to the 
blackout. 

♦ The DPL Stuart- Atlanta 345-kV line, linking 
DPL to AEP and monitored by the PJM reliabil- 
ity coordinator, tripped at 14:02 EDT. This w r as 
the result of a tree contact, and the line 
remained out of service during the entire black- 
out. As explained below, system modeling by 
the investigation team has shown that this out- 
age was not a cause of the subsequent events in 
northern Ohio that led to the blackout. How- 
ever, since the line was not in MISQ’s footprint, 
MISO operators did not monitor the status of 
this line, and did not know that it had gone out 
of service. This led to a data mismatch that pre- 
vented MISO’s state estimator (a key monitoring 
tool) from producing usable results later in the 
day at a lime when system conditions in FE’s 
control area were deteriorating (see details 
below, page 27). 

♦ Eastlake Unit 5 is a 597-MW generating unit 
located just west of Cleveland near Lake Erie. It 
is a major source of reactive power support for 
the Cleveland area. It tripped at 13:31. The 
cause of the trip was that as the Eastlake 5 oper- 
ator sought to increase the unit's reactive power 
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output (Figure 3.5), the unit’s protection system 
detected a failure and tripped the unit off-line. 
The loss of the Eastlake 5 unit did not put the 
grid into an unreliable state — i.e., it was still 
able to withstand safely another contingency. 
However, the loss of the unit required FE to 
import additional power to make up for the loss 
of the unit’s output (540 MW), made voltage 
management in northern Ohio more challeng- 
ing, and gave FE operators less flexibility in 
operating their system (see details below, page 
27). 

Model-Based Analysis of the State 
of the Regional Power System at 
15:05 EDT, Before the Loss of FE’s 
Harding-Chamberlin 345-kV line 

As the first step in modeling the evolution of the 
August 14 blackout, the investigative team estab- 
lished a base case by creating a power flow simula- 
tion for the entire Eastern Interconnection and 
benchmarking it to recorded system conditions at 
15:05 EDT on August 14. The team started with a 
projected summer 2003 power flow case devel- 
oped in the spring of 2003 by the Regional Reli- 
ability Councils to establish guidelines for safe 
operations for the coming summer. The level of 
detail involved in this region-wide study far 
exceeds that normally considered by individual 
control areas and reliability coordinators. It con- 
sists of a detailed representation of more than 
43,000 buses (points at which lines, transformers, 
and/or generators converge), 57,600 transmission 
lines, and all major generating stations across the 
northern U.S. and eastern Canada. The team then 
revised the summer power flow case to match 
recorded generation, demand, and power inter- 
change levels among control areas at 15:05 EDT on 
August 14. The benchmarking consisted of match- 
ing the calculated voltages and line flows to 
recorded observations at more than 1,500 loca- 
tions within the grid. Thousands of hours of effort 
were required to benchmark the model satisfacto- 
rily to observed conditions at 15:05 EDT. 

Once the base case was benchmarked, the team 
ran a contingency analysis that considered' more 
than 800 possible events as points of departure 
from the 15:05 EDT case. None of these contingen- 
cies resulted in a violation of a transmission line 
loading or bus voltage limit prior to the trip of FE’s 


Figure 3.5. MW and MVAr Output from Eastiake 
Unit 5 on August 14 



Harding-Chamberlin 345-kV line. That is, accord- 
ing to these simulations, the system at 15:05 EDT 
was able to be operated safely following the occur- 
rence of any of the tested contingencies. From an 
electrical standpoint, therefore, the Eastern Inter- 
connection was then being operated within all 
established limits and in full compliance with 
NERC’s operating policies. However, after loss of 
the Harding-Chamberlin 345-kV line, the system 
would have exceeded emergency ratings on sev- 
eral lines for two of the contingencies studied. In 
other words, it would no longer be operating in 
compliance with NERC operating policies. 

Conclusion 

Determining that the system was in a reliable 
operational state at 15:05 EDT is extremely signifi- 
cant for understanding the causes ofthe blackout 
It means that none of the electrical conditions on 
the system before 15:05 EDT was a cause of the 
blackout. This eliminates high power flows to 
Canada, unusual system frequencies, low voltages 
earlier in the day or on prior days, and the unavail- 
ability of individual generators or transmission 
lines, either individually or in combination with 
one another, as direct, principal or sole causes of 
the blackout. 

Endnotes 

1 DO&NEKC fact-finding meeting. September 2003, state- 
ment by Mr. Steve Morgan (FE), PR0890803, tines 5-23. 

2 Transmission operator at FE requested the restoration of the 
Avon Substation capacitor bank #2. Example at Channel 3, 
1 3:33:40. 

3 From 13:13 through 13:28, reliability operator at FE called 
nine plant operators to request additional voltage support. 
Examples at Channel 16 , 13:13:18. 13:15:49, 13:16:44, 
13:20:44, 13:22:07, 13:23:24, 13:24:38, 13:26:04, 13:28:40. 
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4. How and Why the Blackout Began 


Summary 

This chapter explains the major events — electri- 
cal, computer, and human — that occurred as the 
blackout evolved on August 14, 2003, and identi- 
fies the causes of the initiation of the blackout. It 
also lists initial findings concerning violations of 
NERC reliability standards. It presents facts col- 
lected by the investigation team and does not offer 
speculative or unconfirmed information or 
hypotheses. Some of the information presented 
here, such as the timing of specific electrical 
events, updates the Sequence of Events 1 released 
earlier by the Task Force. 

The period covered in this chapter begins at 12:15 
Eastern Daylight Time (EDT) on August 14, 2003 
when inaccurate input data rendered MISO’s state 
estimator (a system monitoring tool) ineffective. 
At 13:31 EDT, FE’s Eastlake 5 generation unit trip- 
ped and shut down automatically. Shortly after 
14:14 EDT, the alarm and logging system in FE’s 
control room failed and was not restored until 
after the blackout. After 15:05 EDT, some of FE's 
345-kV transmission lines began tripping out 
because the lines were contacting overgrown trees 
within the lines' righhof-way areas. 

By around 15:46 EDT when FE, MISO and neigh- 
boring utilities had begun to realize that the FE 
system was in jeopardy, the only way that the 
blackout might have been averted would have 
been to drop at least 1,500 to 2,500 MW of load 
around Cleveland and Akron, and at this time the 
amount of load reduction required was increasing 
rapidly. No such effort was made, however, and by 
15:46 EDT it may already have been too late 
regardless of any such effort. After 15:46 EDT, the 
loss of some of FE’s key 345-kV lines in northern 
Ohio caused its underlying network of 138-kV 
lines to begin to fail, leading in turn to the loss of 


FE’s Sammis-Star 345-kV line at 16:06 EDT. The 
chapter concludes with the loss of FE’s Sammis- 
Star line, the event that triggered the uncontrolla- 
ble cascade portion of the blackout sequence. 

The loss of the Sammis-Star line triggered the cas- 
cade because it shut down the 345-kV path into 
northern Ohio from eastern Ohio. Although the 
area around Akron, Ohio was already blacked out 
due to earlier events, most of northern Ohio 
remained interconnected and electricity demand 
was high. This meant that the loss of the heavily 
overloaded Sammis-Star line instantly created 
major and unsustainable burdens on lines in adja- 
cent areas, and the cascade spread rapidly as lines 
and generating units automatically took them- 
selves out of service to avoid physical damage. 

Chapter Organization 

This chapter is divided into several phases that 
correlate to major changes within the FirstEnergy 
system and the surrounding area in the hours 
leading up to the cascade: 

♦ Phase 1: A normal afternoon degrades 

♦ Phase 2: FE's computer failures 

♦ Phase 3: Three FE 345-kV transmission line fail- 
ures and many phone calls 

♦ Phase 4: The collapse of the FE 138-kV system 
and the loss of the Sammis-Star line 

Key events within each phase are summarized in 
Figure 4.1, a timeline of major events in the origin 
of the blackout in Ohio. The discussion that fol- 
lows highlights and explains these significant 
events within each phase and explains how the 
events were related to one another and to the 
cascade. 
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Figure 4.1. Timeline: Start of the Blackout in Ohio 
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Phase 1: 

A Normal Afternoon Degrades: 
12:15 EDT to 14:14 EDT 

Overview of This Phase 

Northern Ohio was experiencing an ordinary 
August afternoon, with loads moderately high to 
serve air conditioning demand. FirstEnergy (FE) 
was importing approximately 2,000 MW into its 
service territory, causing its system to consume 
high levels of reactive power. With two of Cleve- 
land’s active and reactive power production 
anchors already shut down (Davis-Besse and 
Eastlake 4), the loss of the Eastlake 5 unit at 13:31 
further depleted critical voltage support for the 
Cleveland-Akron area. Detailed simulation model- 
ing reveals that the loss of Eastlake 5 was a signifi- 
cant factor in the outage later that afternoon — with 
Eastlake 5 gone, transmission line loadings 
were notably higher and after the loss of FE’s 
Harding-Chamberlin line at 15:05, the system 


eventually became unable to sustain additional 
contingencies without line overloads above emer- 
gency ratings: Had Eastlake 5 remained in service, 
subsequent line loadings would have been lower 
and tripping due to tree contacts may not have 
occurred. Loss of Eastlake 5, however, did not ini- 
tiate the blackout. Subsequent computer failures 
leading to the loss of situational awareness in FE’s 
control room and the loss of key FE transmission 
lines due to contacts with trees were the most 
important causes. 

At 14:02 EDT, Dayton Power & Light’s (DPL) Stu- 
art-Atlanta 345-kV line tripped off-line due to a 
tree contact. This line had no direct electrical 
effect on FE’s system — but it did affect MISO’s per- 
formance as reliability coordinator, even though 
PJM is the reliability coordinator for the DPL line. 
One of MISO’s primary system condition evalua- 
tion tools, its state estimator, was unable to assess 
system conditions for most of the period between 
12:37 EDT and 15:34 EDT, due to a combination of 
human error and the effect of the loss of DPL’s 


22 


••> ILS.-Canada Power System Outage Task Force ❖ Causes of the August 14th Blackout 

















237 



‘ v'. 


Cause 3: 


Cause 1: 


Cause 2: 


iware earlier of important line 


• lacked joint procedures or 
'ben and how to coordinate a 


security limit violation observed by one of 
them in the other’s area due to a contingency ; 
near their common boundary. (See page 38.) 

In the pages below, sections that relate to par- 


lar causes are denoted with (he following 


Group 3: Failure of tlie interconnected grid’s 
reliability organizations to provide effective 
diagnostic support. In particular: 


Stuart-Atlanta line on other MISO lines as 
reflected in the state estimator’s calculations. 
Without an effective state estimator, MISO was 
unable to perform contingency analyses of genera- 
tion and line losses within its reliability zone. 
Therefore, through 15:34 EDT MISO could not 
determine that with Eastlake 5 down, other trans- 
mission lines would overload if FE lost a major 
transmission line, and could not issue appropriate 
warnings and operational instructions. 

In the investigation interviews, all utilities, con- 
trol area operators, and reliability coordinators 


indicated that the morning of August 14 was a rea- 
sonably typical day. FE managers referred to it as 
peak load conditions on a less than peak load 
day. 2 Dispatchers consistently said that while 
voltages were low, they were consistent with his- 
torical voltages. 3 Throughout the morning and 
early afternoon of August 14, FE reported a grow- 
ing need for voltage support in the upper Midwest. 

The FE reliability operator was concerned about 
low voltage conditions on the FE system as early 
as 13:13 EDT. He asked for voltage support (i.e., 
increased reactive power output) from FE's 
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Figure 4.2. Timeline Phase 1 



interconnected generators. Plants were operating 
in automatic voltage control mode (reacting to sys- 
tem voltage conditions and needs rather than con- 
stant reactive power output). As directed in FE's 
Manual of Operations, 4 the FE reliability operator 
began to call plant operators to ask for additional 
voltage support from their units. He noted to most 
of them that system voltages were sagging “all 
over.” Several mentioned that they were already at 
or near their reactive output limits. None were 
asked to reduce their active power output to be 
able to produce more reactive output. He called 
the Sammis plant at 13:13 EDT, West Lorain at 
13:15 EDT, Eastlake at 13:16 EDT, made three 
calls to unidentified plants between 13:20 EDT 
and 13:23 EDT, a “Unit 9” at 13:24 EDT, and two 
more at 13:26 EDT and 13:28 EDT. 5 The operators 
worked to get shunt capacitors at Avon that were 
out of service restored to support voltage. 6 

Following the loss of Eastlake 5 at 13:31 EDT, FE’s 
operators’ concern about voltage levels was 
heightened. They called Bayshore at 13:41 EDT 
and Perry at 13:43 EDT to ask the plants for more 
voltage support. Again, while there was substan- 
tial effort to support voltages in the Ohio area, 
First Energy personnel characterized the condi- 
tions as not being unusual for a peak load day, 
although this was not an all-time (or record) peak 
load day. 

Key Phase 1 Events 

1 A) 12:15 EDT to 16:04 EDT: MISO's state estima- 
tor software solution was compromised, and 
MISO’s single contingency reliability assess- 
ment became unavailable. 


IB) 13:31:34 EDT : Eastlake Unit 5 generation trip- 
ped in northern Ohio. 

IC) 14:02 EDT: Stuart-Atlanta 345-kV transmis- 
sion line tripped in southern Ohio. 

1A) MISO’s State Estimator Was Turned Off: 
12:15 EDT to 16:04 EDT 

It is common for reliability coordinators and con- 
trol areas to use a tool called a state estimator (SE) 
to improve the accuracy of the raw sampled data 
they have for the electric system by mathemati- 
cally processing raw data to make it consistent 
with the electrical system model. The resulting 
information on equipment voltages and loadings 
is used in software tools such as real time contin- 
gency analysis (RTCA) to simulate various condi- 
tions and outages to evaluate the reliability of the 
power system. The RTCA tool ismsed to alert oper- 
ators if the system is operating insecurely; it can 
be run either on a regular schedule (e.g., every 5 
minutes), when triggered by some system event 
(e.g., the loss of a power plant or transmission 
line), or when initiated by an operator. MISO usu- 
ally runs the SE every 5 minutes, and the RTCA 
less frequently. If the model does not have accu- 
rate and timely information about key pieces of 
system equipment or if key input data are wrong, 
the state estimator may be unable to reach a solu- 
tion or it will reach a solution that is labeled as 
having a high degree of error. MISO considers its 
SE and RTCA tools to be still under development 
and not fully mature. 

On August 14 at about 12:15 EDT, MISO’s state 
estimator produced a solution with a high mis- 
match (outside the bounds of acceptable error). 
This was traced to an outage of Cinergy’s 
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Initial Findings: Violations of NEftC Reliability Standards 

Note: These are initial findings and subject to thecausc{s)ofOPBliAT 

further review by NERC. Additional violations tions. This injormatii 

may be identified. both real time and prei 

Violation Number 1. Following the outage of the appropriate corrective 

Chamberlin -Harding 345-kV line, FE did not take Violation Number 4. JF1 


hollowing a contingency or other event that 
results in an OPERATING SKCimnY UMIT viola- 
tion, the CONTROL AREA shall return its trans- 
mission system to within OPERATING SECURITY 
LIMITS soon as possible, but no longer than 30 
minutes. 

, . ..V. - - ■ _ .. - i 

Violation Number 2. FE did not notify other sys- 
tems of an impending system emergency. b 

Reference: NERC Operating Policy 5: 

Notifying other systems. A system shall inform 
other systems in their Region or subregion, 
through predetermined communication paths, 
whenever the following situations are antici- 
pated or arise: 

System is burdening others. The system ’s con- 
dition is burdening other systems or reducing 
the reliability of the Interconnection. 

Lack of single contingency coverage. The sys- 
tem’s line loadings and voltage! reactive levels 
are such that a single contingency could 
threaten the reliability of the Interconnection. 

Violation Number 3. FE’s state estimation/con- 
tingenny analysis tools were not used to assess 
the system conditions. 0 

Reference: NERC Operating Policy 5: 

Sufficient information and analysis tools shall be 
provided to the SYSTEM OPERATOR to determine 


Violation Number 5. MISO did not notify other 
reliability coordinators of potential problems. 6 

Reference: NERC Operating Policy 9; 

Notify reliability Coordinators of potential 


problems. The RELIABILITY COORDINATOR who 
foresees a transmission prpblem within his 
Reliability Area shall issue an alert to all 
CONTROL AREAS and Transmission Providers hi 
his Reliability area, and all reliability 
Coordinators within the Interconnection via 
the RCIS without delay. 

( continued on following page) 

“Investigation team modeling showed that following the loss qf the Ghamberlin-Harding 345-kV line the system was beyond its 
OPERATING SECURITY Limit; i.e., the loss of the next most severe contingency would have resulted in other lines exceeding their 
emergency limits. Blackout causes 1 A, IB. IE. 

’’OOK on-site interviews; comparative review of FEand MTSO phone transcripts of 14 /August; no calls found of FE declaring an 
emergency to MISO in either set of transcripts. Blackout causes 1 A, IB, II), IE. 
c !)OE on-site interviews; Mr. Morgan. September 8 and 9 transcripts. 

**Site visit by interviewers from Operations Team. 

e MJSO site visit and DOE interviews; Oct 1-3 Newark meetings, nsl00303.pdf; Harzey-Caulcy conversation, pages 1 1 1-119; 
blackout cause 3D. 
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Initial Findings: Violations of NERC Reliability Standards (Continued) 


Violation Number 6. MISO cUd not bave ade- 
quate monitbriiig capability 
Reference: NERC Operating Policy 9, Appendix 
9D: 

Adequate facilities Must have the facilities to 
perform their responsibilities, including: 

♦ Detailed monitoring capability of the 
'y~', muABiWY Aim and sufficient monitoring 


capability of the surrounding R$UABILlrii 
AREAS to ensure potential security violations^ 
are identified. .A 

Continuous monitoring of Reliability Area A 
Must ensure that its REIMBJIIIY ABEA ofresp 0 nA\ 
sibility is continuously and adequately nroniA 
tored. This includes the provisions for baclaipti 
facilities. 


tDOE interview? and Operations Team site visit. Oct 1-3 Newark meetings, nsl003Q3.pdf; Harzey-Cauley conversation, pages 
ill-119; blackout causes 3A, 3B. 3C. . '' c- ? 
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Bloomington-Denois Creek 230-kV line — al- 
though it was out of service, its status was not 
updated in MISO’s state estimator. Line status 
information within MISO’s reliability coordina- 
tion area is transmitted to MISO by the ECAR data 
network or direct links and intended to be auto- 
matically linked to the SE. This requires coordi- 
nated data naming as well as instructions that link 
tire data to the tools. For this line, the automatic 
linkage of line status to the state estimator had not 
yet been established (this is an ongoing project at 
MISO). The line status was corrected and MISO’s 
analyst obtained a good SE solution at 13:00 EDT 
and an RTCA solution at 13:07 EDT, but to trou- 
bleshoot this problem he had turned off the auto- 
matic trigger that runs the state estimator every 
five minutes. After fixing the problem he forgot to 
re-enable it, so although he had successfully run 
the SE and RTCA manually to reach a set of correct 
system analyses, the tools were not returned to 
normal automatic operation. Thinking the system 
had been successfully restored, the analyst went 
to lunch. 

The fact that the state estimator 
was not running automatically on 
its regular 5-minute schedule was 
discovered about 14:40 EDT. The 
automatic trigger was re-enabled 
but again the state estimator failed to solve suc- 
cessfully. This time investigation identified the 
Stuart-Atlanta 345-kV line outage (14:02 EDT) to 
be the likely cause. 7 This line is jointly owned by 
Dayton Power and Light and AEP and is moni- 
tored by Dayton Power and Light and is under 
PJM’s reliability umbrella rather than MISO’s. 
Even though it affects electrical flows within 
MISO, its status had not been automatically linked 
to MISO’s SE. 

The discrepancy between actual measured system 
flows (with Stuart-Atlanta off-line) and the MISO 
model (which assumed Stuart-Atlanta on-line) 
prevented the state estimator from solving 
correctly. At 15:09 EDT, when informed by the 
system engineer that the Stuart-Atlanta line 
appeared to be the problem, the MISO operator 
said (mistakenly) that this line was in service. The 
system engineer then tried unsuccessfully to 
reach a solution with the Stuart-Atlanta line mod- 
eled as in service until approximately 15:29 EDT, 
when the MISO operator called PJM to verify the 
correct status. After they determined that Stu- 
art-Atlanta had tripped, they updated the state 
estimator and it solved successfully. The RTCA 
was then run manually and solved successfully at 


15:41 EDT. MISO’s state estimator and contin- 
gency analysis were back under full automatic 
operation and solving effectively by 16:04 EDT, 
about two minutes before the initiation of the 
cascade. 

In summary, the MISO state estimator and real 
time contingency analysis tools were effectively 
out of service between 12:15 EDT and 16:04 EDT. 
This prevented MISO from promptly performing 
precontingency “early warning” assessments of 
power system reliability over the afternoon of 
August 14. 

IB) Eastlake Unit 5 Tripped: 13:31 EDT 

Eastlake Unit 5 (rated at 597 MW) is in northern 
Ohio along the southern shore of Lake Erie, con- 
nected to FE’s 345-kV transmission system (Figure 
4.3). The Cleveland and Akron loads are generally 
supported by generation from a combination of 
the Eastlake and Davis-Besse units, along with sig- 
nificant imports, particularly from 9,100 MW of 
generation located along the Ohio and Pennsylva- 
nia border. The unavailability of Eastlake 4 and 
Davis-Besse meant that FE had to import more 
energy into the Cleveland area (either from its own 
plants or from or through neighboring utilities) to 
support its load. 

When Eastlake 5 dropped off-line, flows caused by 
replacement power transfers and the associated 
reactive power to support the imports to the local 
area contributed to the additional line loadings in 
the region. At 15:00 EDT on August 14, FE’s load 
was approximately 12,080 MW. They were 
importing about 2,575 MW, 21% of their total. 
With this high level of imports, FE's system reac- 
tive power needs rose further. Investigation team 
modeling indicates that at about 15:00 EDT, FE’s 


Figure 4.3. Eastlake Unit 5 
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system was consuming so much reactive power 
that it was a net importer, bringing in about 132 
MVAr. 

The investigation team’s system simulations indi- 
cate that the loss of Eastlake 5 was a critical step in 
the sequence of events. Contingency analysis sim- 
ulation of the conditions following the loss of the 
Harding-Chamberlin 345-kV circuit at 15:05 EDT 
showed that the system would be unable to sus- 
tain some contingencies without line overloads 
above emergency ratings. However, when Eastlake 
5 was modeled as in service and fully available in 
those simulations, all overloads above emergency 
limits were eliminated even with the loss of 
Harding-Chamberlin. 

jggggW tt FE did not perform a contingency 
analysis after the loss of Eastlake 
5 at 13:31 EDT to determine 
whether the loss of further lines 
or plants would put their system 
at risk. FE also did not perform a contingency anal- 
ysis after the loss of Harding-Chamberlin at 15:05 
EDT (in part because they did not know that it had 
tripped out of service), nor does the utility rou- 
tinely conduct such studies. 8 Thus FE did not dis- 
cover that their system was no longer in an N-l 
secure state at 15:05 EDT, and that operator action 
was needed to remedy the situation. 

1C) Stuart-Atlanta 345-kV Line Tripped: 

14:02 EDT 

The Stuart-Atlanta 345-kV trans- 
mission line is in the control area 
of Dayton Power and Light. 9 At 
14:02 EDT the line tripped due to 
contact with a tree, causing a 
short circuit to ground, and locked out. Investiga- 
tion team modeling reveals that the loss of DPL’s 
Stuart-Atlanta line had no significant electrical 
effect on power flows and voltages in the FE area. 
The team examined the security of FE’s system, 
testing power flows and voltage levels with the 
combination of plant and line outages that evolved 
on the afternoon of August 14. This analysis 
shows that the availability or unavailability of the 
Stuart-Atlanta 345-kV line did not change the 
capability or performance of FE’s system or affect 
any line loadings within the FE system, either 
immediately after its trip or later that afternoon. 
Again, the only reason why Stuart-Atlanta matters 
to the blackout is because it contributed to the fail- 
ure of MISO’s state estimator to operate effec- 
tively, so MISO could not fully identify FE’s 
precarious system conditions until 16:04 EDT. 


Phase 2: 

FE’s Computer Failures: 

14:14 EDT to 15:59 EDT 

Overview of This Phase 

Starting around 14:14 EDT, FE’s control room 
operators lost the alarm function that provided 
audible and visual indications when a significant 
piece of equipment changed from an acceptable to 
problematic condition. Shortly thereafter, the 
EMS system lost a number of its remote control 
consoles. Next it lost the primary server computer 
that was hosting the alarm function, and then the 
backup server such that all functions that were 
being supported on these servers were stopped at 
14:54 EDT. However, for over an hour no one in 
FE’s control room grasped that their computer sys- 
tems were not operating properly, even though 
FE's Information Technology support staff knew 
of the problems and were working to solve them, 
and the absence of alarms and other symptoms 
offered many clues to the operators of the EMS 
system’s impaired state. Thus, without a function- 
ing EMS or the knowledge that it had failed, FE’s 
system operators remained unaware that their 
electrical system condition was beginning to 
degrade. Unknowingly, they used the outdated 
system condition information they did have to dis- 
count information from others about growing sys- 
tem problems. 

Key Events in This Phase 

2A) 14:14 EDT: FE alarm and logging software 
failed. Neither FE’s control room operators 
nor FE’s IT EMS support personnel were 
aware of the alarm failure. 

2B) 14:20 EDT: Several FE remote location con- 
soles failed. FE Information Technology (IT) 
engineer was computer auto-paged. 

2C) 14:27:16 EDT: Star-South Canton 345-kV 
transmission line tripped and successfully 
reclosed. 

2D) 14:32 EDT: AEP called FE control room about 
AEP indication of Star-South Canton 345-kV 
line trip and reclosure. FE had no alarm or log 
of this line trip. 

2E) 14:41 EDT: The primary FE control system 
server hosting the alarm function failed. Its 
applications and functions were passed over 
to a backup computer. FE's IT engineer was 
auto-paged. 
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Figure 4.4. Timeline Phase 2 



2F) 14:54 EDT: The FE back-up computer failed 
and all functions that were running on it 
stopped. FE’s IT engineer was auto-paged. 

Failure of FE’s Alarm System 

. pg» s com p ll t er SCADA alarm and 
logging software failed sometime 
shortly after 14:14 EDT (the last 
time that a valid alarm came in). 
After that time, the FE control 
room consoles did not receive any further alarms 
nor were there any alarms being printed or posted 
on the EMS's alarm logging facilities. Power sys- 
tem operators rely heavily on audible and 
on-screen alarms, plus alarm logs, to reveal any 
significant changes in their system’s conditions. 
After 14:14 EDT on August 14, FE’s operators were 
working under a significant handicap without 
these tools. However, they were in further jeop- 
ardy because they did not know that they were 
operating without alarms, so that they did not real- 
ize that system conditions were changing. 

Alarms are a critical function of an EMS, and 
EMS-generated alarms are the fundamental means 
by which system operators identify events on the 
power system that need their attention. Without 
alarms, events indicating one or more significant 
system changes can occur but remain undetected 
by the operator. If an EMS’s alarms are absent, but 
operators are aware of the situation and the 
remainder of the EMS’s functions are intact, the 
operators can potentially continue to use the EMS 
to monitor and exercise control of their power sys- 
tem. In such circumstances, the operators would 
have to do so via repetitive, continuous manual 
scanning of numerous data and status points 


located within the multitude of individual dis- 
plays available within their EMS. Further, it 
would be difficult for the operator to identify 
quickly the most relevant of the many screens 
available. 

Although the alarm processing function of FE’s 
EMS failed, the remainder of that system generally 
continued to collect valid real-time status infor- 
mation and measurements about FE’s power sys- 
tem, and continued to have supervisory control 
over the FE system. The EMS also continued to 
send its normal and expected collection of infor- 
mation on to other monitoring points and authori- 
ties, including MISO and AEP. Thus these entities 
continued to receive accurate information about 
the status and condition of FE's power system 
even past the point when FE’s EMS alarms failed. 
FE’s operators were unaware that in this situation 
they needed to manually and more closely moni- 
tor and interpret the SCADA information they 
were receiving. Continuing on in the belief that 
their system was satisfactory and lacking any 
alarms from their EMS to the contrary, FE control 
room operators were subsequently surprised 
when they began receiving telephone calls from 
other locations and information sources — MISO, 
AEP, PJM, and FE field operations staff— -who 
offered information on the status of FE’s transmis- 
sion facilities that conflicted with FE’s system 
operators’ understanding of the situation. 

Analysis of the alarm problem performed by FE 
suggests that the alarm process essentially 
“stalled” while processing an alarm event, such 
that the process began to run in a manner that 
failed to complete the processing of that alarm or 
produce any other valid output (alarms). In the 
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meantime, new inputs-— system condition data 
that needed to be reviewed for possible alarms — 
built up in and then overflowed the process’ input 
buffers. 10 

toss of Remote EMS Terminals. Between 14:20 
EDT and 14:25 EDT, some of FE's remote control 
terminals in substations ceased operation. FE has 
advised the investigation team that it believes this 
occurred because the data feeding into those ter- 
minals started “queuing” and overloading the ter- 
minals* buffers. FE’s system operators did not 
learn about this failure until 14:36 EDT, when a 
technician at one of the sites noticed the terminal 
was not working after he came in on the 15:00 
shift, and called the main control room to report 
the problem. As remote terminals failed, each 
triggered an automatic page to FE’s Information 


Technology (IT) staff. 11 The investigation team 
has not determined why some terminals failed 
whereas others did not. Transcripts indicate that 
data links to the remote sites were down as well. 12 

EMS Server Failures. FE’s EMS system includes 
several server nodes that perform the higher func- 
tions of the EMS. Although any one of them can 
host all of the functions, FE's normal system con- 
figuration is to have a number of host subsets of 
the applications, with one server remaining in a 
“hot-standby” mode as a backup to the others 
should any fail. At 14:41 EDT, the primary server 
hosting the EMS alarm processing application 
failed, due either to the stalling of the alarm 
application, “queuing” to the remote terminals, 
or some combination of the two. Following 
preprogrammed instructions, the alarm system 
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application and all other EMS software running on 
the first server automatically transferred (“failed- 
over”) onto the back-up server. However, because 
the alarm application moved intact onto the 
backup while still stalled and ineffective, the 
backup server failed 13 minutes later, at 14:54 
EOT. Accordingly, all of the EMS applications on 
these two servers stopped running. 

The concurrent loss of both EMS servers appar- 
ently caused several new problems for FE’s EMS 
and the operators who used it. Tests run during 
FE’s after-the-fact analysis of the alarm failure 
event indicate that a concurrent absence of these 
servers can significantly slow down the rate at 
which the EMS system puts new — or refreshes 
existing — displays on operators' computer con- 
soles. Thus at times on August 14th, operators’ 
screen refresh rates — the rate at which new infor- 
mation and displays are painted onto the com- 
puter screen, normally 1 to 3 seconds — slowed to 
as long as 59 seconds per screen. Since FE opera- 
tors have numerous information screen options, 
and one or more screens are commonly “nested” as 
sub-screens to one or more top level screens, oper- 
ators’ ability to view, understand and operate their 
system through the EMS would have slowed to a 
frustrating crawl. 13 This situation may have 
occurred between 14:54 EDT and 15:08 EDT when 
both servers failed, and again between 15:46 EDT 
and 15:59 EDT while FE’s IT personnel attempted 
to reboot both servers to remedy the alarm 
problem. 

Loss of the first server caused an auto-page to be 
issued to alert FE’s-EMS IT support personnel to 
th e-problem. When the back-up server failed, it 
too sent an auto-page to FE’s IT staff. At 15:08 
EDT, IT staffers completed a “warm reboot” 
(restart) of the primary server. Startup diagnostics 
monitored during that reboot verified that the 
computer and all expected processes were run- 
ning; accordingly, FE's IT staff believed that they 
had successfully restarted the node and all the 
processes it was hosting. However, although the 
server and its applications were again runn i ng, the 
alarm system remained frozen and non-func- 
tional, even on the restarted computer. The IT staff 
did not confirm that the alarm system was again 
working properly with the control room operators. 

Another casualty of the loss of both servers was 
the Automatic Generation Control (AGC) function 
hosted on those computers. Loss of AGC meant 
that FE’s operators could not run affiliated 
power plants on pre-set programs to respond 


automatically to meet FE’s system load and inter- 
change obligations. Although the AGC did not 
work from 14:54 EDT to 15:08 EDT and 15:46 EDT 
to 15:59 EDT (periods when both servers were 
down), this loss of function does not appear to 
have had any effect on the blackout. 

The concurrent loss of the EMS servers also 
caused the failure of FE's strip chart function. 
There are many strip charts in the FE Reliability 
Operator control room driven by the EMS comput- 
ers, showing a variety of system conditions, 
including raw ACE (Area Control Error), FE Sys- 
tem Load, and Sammis-South Canton and South 
Canton-Star loading. These charts are visible in 
the reliability operator control room. The chart 
printers continued to scroll but because the under- 
lying computer system was locked up the chart 
pens showed only the last valid measurement 
recorded, without any variation from that mea- 
surement as time progressed; i.e. the charts 
“flat-lined.” There is no indication that any opera- 
tors noticed or reported the failed operation of the 
charts. 14 The few charts fed by direct analog 
telemetry, rather than the EMS system, showed 
primarily frequency data, and remained available 
throughout the afternoon of August 14. These 
yield little useful system information for opera- 
tional purposes. 

FE’s Area Control Error (ACE), the primary control 
signal used to adjust generators and imports to 
match load obligations, did not function between 
14:54 EDT and 15:08 EDT and later between 15:46 
EDT and 15:59 EDT, when the two servers were 
down. This meant that generators were not con- 
trolled-during these periods to meet FE’s load and 
interchange obligations (except from 15:00 EDT to 
15:09 EDT when control was switched to a backup 
controller). There were no apparent negative 
impacts due to this failure. It has not been estab- 
lished how loss of the primary generation control 
signal was identified or if any discussions 
occurred with respect to the computer system's 
operational status. 15 

EMS System History. The EMS in service at FE's 
Ohio control center is a GE Harris (now GE Net- 
work Systems) XA21 system. It was initially 
brought into service in 1995. Other than the appli- 
cation of minor software fixes or patches typically 
encountered in the ongoing maintenance and sup- 
port of such a system, the last major updates or 
revisions to this EMS were implemented in 1998. 
On August 14 the system was not running the 
most current release of the XA21 software. FE had 
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decided well before August 14 to replace it with 
one from another vendor. 

FE personnel told the investigation team that the 
alarm processing application had failed on occa- 
sions prior to August 14, leading to loss of the 
alarming of system conditions and events for FE’s 
operators. 10 However, FE said that the mode and 
behavior of this particular failure event were both 
first time occurrences and ones which, at the time, 
FE’s IT personnel neither recognized nor knew 
how to correct. FE staff told investigators that it 
was only during a post-outage support call with 
GE late on 14 August that FE and GE determined 
that the only available course of action to correct 
the alarm problem was a “cold reboot” 17 of FE's 
overall XA21 system. In interviews immediately 
after the blackout, FE IT personnel indicated that 
they discussed a cold reboot of the XA21 system 
with control room operators after they were told of 
the alarm problem at 15:42 EDT, but decided not 
to take such action because operators considered 


power system conditions precarious, were con- 
cerned about the length of time that the reboot 
might take to complete, and understood that a cold 
boot would leave them with even less EMS sup- 
port until it was completed . 18 

Clues to the EMS Problems. There is an entry in 
FE’s western desk operator’s log at 14:14 EDT 
referring to the loss of alarms, but it is not clear 
whether that entry was made at that time or subse- 
quently, referring back to the last known alarm. 
There is no indication that the operator mentioned 
the problem to other control room staff and super- 
visors or to FE’s IT staff. 

The first clear hint to FE control room staff of any 
computer problems occurred at 14:19 EDT when a 
caller and an FE control room operator discussed 
the fact that three sub-transmission center 
dial-ups had failed. 19 At 14:25 EDT, a control 
room operator talked with a caller about the fail- 
ure of these three remote terminals. 20 The next 
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hint came at 14:32 EDT, when FE scheduling staff 
spoke about having made schedule changes to 
update the EMS pages, but that the totals did not 
update. 21 

Although FE’s IT staff would have been aware that 
concurrent loss of its servers would mean the loss 
of alarm processing on the EMS, the investigation 
team has found no indication that the IT staff 
informed the control room staff either when they 
began work on the servers at 14:54 EDT, or when 
they completed the primary server restart at 15:08 
EDT. At 15:42 EDT, the IT staff were first told of 
the alarm problem by a control room operator; FE 
has stated to investigators that their IT staff had 
been unaware before then that the alarm process- 
ing sub-system of the EMS was not working. 

Without the EMS systems, the 
only remaining ways to monitor 
system conditions would have 
been through telephone calls and 
direct analog telemetry. FE con- 
trol room personnel did not realize that alarm 
processing on their EMS was not working and, 
subsequently, did not monitor other available 
telemetry. 

During the afternoon of August 
14, FE operators talked to their 
field personnel, MISO, PJM {con- 
cerning an adjoining system in 
PJM’s reliability coordination 
region), adjoining systems (such as AEP), and cus- 
tomers. The FE operators received pertinent infor- 
mation from all these sources, but did not grasp 
some key information about the system from the 
clues offered. This pertinent information included 
calls such as that from FE’s eastern control center 
where they were asking about possible line trips, 
FE Perry nuclear plant calls regarding what looked 
like near-line trips, AEP calling about their end of 
the Star-South Canton line tripping, and MISO 
and PJM calling about possible line overloads. 
Without a functioning alarm system, the FE con- 
trol area operators failed to detect the tripping of 
electrical facilities essential to maintain the secu- 
rity of their control area. Unaware of the loss of 
alarms and a limited EMS, they made no alternate 
arrangements to monitor the system. When AEP 
identified a circuit trip and reclosure on a 345-kV 
line, the FE operator dismissed the information 
as either not accurate or not relevant to his sys- 
tem, without following up on the discrepancy 
between the AEP event and the information from 
his own tools. There was no subsequent verifica- 
tion of conditions with their MISO reliability 



coordinator. Only after AEP notified FE that a 
345-kV circuit had tripped and locked out did the 
FE control area operator compare tills information 
to the breaker statuses for their station. FE failed to 
inform immediately its reliability coordinator and 
adjacent control areas when they became aware 
that system conditions had changed due to 
unscheduled equipment outages that might affect 
other control areas. 

Phase 3: 

Three FE 345-kV 
Transmission Line Failures 
and Many Phone Calls: 

15:05 EDT to 15:57 EDT 

Overview of This Phase 

From 15:05:41 EDT to 15:41:35 EDT, three 345-kV 
lines failed with power flows at or below each 
transmission line’s emergency rating. Each was 
the result of a contact between a line and a tree 
that had grown so tall that, over a period of years, 
it encroached into the required clearance height 
for the line. As each line failed, its outage 
increased the loading on the remaining lines 
{Figure 4.5). As each of the transmission lines 
failed, and power flows shifted to other transmis- 
sion paths, voltages on the rest of FE’s system 
degraded further {Figure 4.6). 

Key Phase 3 Events 

3A) 15:05:41 EDT: Harding-Chamberlin 345-kV 
line tripped. 

3B) 15:31-33 EDT: MISO called PJM to determine 
if PJM had seen the Stuart- Atlanta 345-kV 
line outage. PJM confirmed Stuart- Atlanta 
was out. 


Figure 4.5. FirstEnergy 345-kV Line Flows 



Time - EDT 
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3C) 15:32:03 EDT: Hanna-Juniper 345-kV line 
tripped. 

3D) 15:35 EDT: AEP asked PJM to begin work on a 
3 5 0 -M W TLR to relieve overloading on the 
Star-South Canton line, not knowing the 
Hanna-Juniper 345-kV line had already trip- 
ped at 15:32 EDT. 

3E) 15:36 EDT: MISO called FE regarding 
post-contingency overload on Star-Juniper 
345-kV line for the contingency loss of the 
Hanna-Juniper 345-kV line, unaware at the 
start of the call that Hanna-Juniper had 
already tripped. 

3F) 15:41:33-41 EDT: Star-South Canton 345-kV 
tripped, reclosed, tripped again at 15:41 EDT 
and remained out of. service, all while AEP 
and PJM were discussing TLR relief options 
(event 3D). 


Figure 4,6. Voltages on FirstEnergy’s 345-kV Lines: 


Impacts of Line Trips 



Transmission lines are designed with the expecta- 
tion that they will sag lower when they are hotter. 
The transmission line gels hotter with heavier line 
loading and under higher ambient temperatures, 
so towers and conductors are designed to be tall 
enough and conductors pulled lightly enough to 
accommodate expected sagging. 

A short-circuit occurred on the Harding-Cham- 
berlin 345-kV line due to a contact between the 
line conductor and a tree. This line failed with 
power flow at only 43.5% of its normal and emer- 
gency line rating. Incremental line current and 
temperature increases, escalated by the loss of 
Harding-Chamberlin, caused enough sag on the 
Hanna-Juniper line that it contacted a tree and 
faulted with power flow at 87.5% of its normal 
and emergency line ratings Star-South Canton 
contacted a tree three times between 14:27:15 EDT 
and 15:41:33 EDT, opening and reclosing each 
lime before finally locking out while loaded at 
93.2% of its emergency rating at 15:42:35 EDT. 

Overgrown trees, as opposed to 
excessive conductor sag, caused 
each of these faults. While sag 
may have contributed to these 
events, these incidents occurred 
because the trees grew too tall and encroached 
into the space below the line which is intended 
to be clear of any objects, not because the lines 
sagged into short trees. Because the trees were so 
tall (as discussed below), each of these lines 
faulted under system conditions well within spec- 
ified operating parameters. The investigation team 
found field evidence of tree contact at all three 
locations, although Hanna-Juniper is the only 
one with a confirmed sighting for the August 14 
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To be sure that the evidence of tree/line contacts 
and tree remains found at each site was linked to 
the events of August 14, the team looked at 
whether these lines had any prior history of out- 
ages in preceding months or years that might have 
resulted in the burn marks, debarking, and other 
vegetative evidence of line contacts, The record 
establishes that there were no prior sustained out- 
ages known to be caused by trees for these lines in 
2001, 2002 and 2003. 22 

Like most transmission owners, FE patrols its lines 
regularly, flying over each transmission line twice 
a year to check on the condition of the rights- 
of-way. Notes from fly-overs in 2001 and 2002 
indicate that the examiners saw a significant num- 
ber of trees and brush that needed clearing or trim- 
ming along many FE transmission lines. 
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3 A) FE’s Harding-Chamberlin 345-kV Line 
Tripped: 15:05 EDT 

At 15:05:41 EDT, FE’s Harding- 
Chamberlin line (Figure 4.8} 
tripped and locked out while 
loaded at 43.5% of its normal and 
emergency rating. The investiga- 
tion team has examined the relay data for this trip, 
identified the geographic location of the fault, and 
determined that the relay data match the classic 
“signature” pattern for a tree/line short circuit to 
ground fault. Going to the fault location deter- 
mined from the relay data, the field team found 
the remains of trees and brush. At this location, 
conductor height measured 46 feet 7 inches, while 
the height of the felled tree measured 42 feet; how- 
ever, portions of the tree had been removed from 
the site. This means that while it is difficult to 
determine the exact height of the line contact, the 
measured height is a minimum and the actual con- 
tact was likely 3 to 4 feet higher than estimated 
here. Burn marks were observed 35 feet 8 inches 
up the tree, and the crown of this tree was at least 6 
feet taller than the observed bum marks. The tree 
showed evidence of fault current damage. 23 

When the Harding-Chamberlin line locked out, 
the loss of this 345-kV path caused the remaining 
three southern 345-kV lines into Cleveland to pick 
up more load, with Hanna-Juniper picking up 
the most. The Harding-Chamberlin outage also 
caused more power to flow through the underly- 
ing 138-kV system. 

MISO did not discover that Har- 
ding-Chamberlin had tripped 
until after the blackout, when 
MISO reviewed the breaker 
operation log that evening. FE 
indicates that it discovered the line was out while 
investigating system conditions in response 
MISO's call at 15:36 EDT, when MISO told FE that 
MISO’s flowgate monitoring tool showed a Star- 
Juniper line overload following a contingency loss 
of Hanna-Juniper; 24 however, tire investigation 
team has found no evidence within the control 
room logs or transcripts to show that FE knew of 
the Harding-Chamberlin line failure until after the 
blackout. 

Harding-Chamberlin was not one 
of the flowgates that MISO moni- 
tored as a key transmission loca- 
tion, so the reliability coordinator 
was unaware when FE’s first 345-kV line failed. 
Although MISO received SC AD A input of the 



Figure 4.8. Harding-Chamberlin 345-kV Line 



line’s status change, this was presented to MISO 
operators as breaker status changes rather than a 
line failure. Because their EMS system topology 
processor had not yet been linked to recognize line 
failures, it did not connect the breaker information 
to the loss of a transmission line. Thus, MISO’s 
operators did not recognize the Harding- 
Chamberlin trip as a significant contingency event 
and could not advise FE regarding the event or its 
consequences. Further, without its state estimator 
and associated contingency analyses, MISO was 
unable to identify potential overloads that would 
occur due to various line or equipment outages. 
Accordingly, when the Harding-Chamberlin 
345-kV line tripped at 15:05 EDT, the state estima- 
tor did not produce results and could not predict 
an overload if the Hanna-Juniper 345-kV line were 
to fail. 25 


3C)£E’s Hanna-Juniper 345-kV Line Tripped: 
15:32 EDT 



At 15:32:03 EDT the Hanna- 
Juniper line (Figure 4.9) tripped 
and locked out. A tree-trimming 
crew was working nearby and 
observed the tree/line contact. 
The tree contact occurred on tire South phase, 
which is lower than the center phase due to 
construction design. Although little evidence re- 
mained of the tree during the field team’s visit in 
October, the team observed a tree stump 14 inches 
in diameter at its ground line and talked to an indi- 
vidual who witnessed the contact on August 14. 26 


FE provided photographs that clearly indicate that 
the tree was of excessive height. Surrounding trees 
were 18 inches in diameter at ground line and 60 
feet in height (not near lines). Other sites at this 
location had numerous (at least 20) trees in this 
right-of-way. 
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Figure 4.10. Cause of the Hanna- Juniper Line Loss 


Figure 4.9. Hanna-Juniper 345-kV Line 


• (Hannah 


Area oftpetail/i 


This August 14 photo shows the tree that caused the loss of 
the Hanna-Juniper line (tallest tree in photo). Other 345-kV 
conductors and shield wires can be seen in the background. 
Photo by Nelson Tree. 
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Hanna-Juniper was loaded at 87.5% of its normal 
and emergency rating when it tripped. With this 
line open, almost 1,000 MVA had to find a new 
path to reach its load in Cleveland. Loading on the 
remaining two 345-kV lines increased, with 
Star-Juniper taking the bulk of the power. This 
caused Star-South Canton’s loading to rise above 
its normal but within its emergency rating and 
pushed more power onto the 138-kV system. 
Flows west into Michigan decreased slightly and 
voltages declined somewhat in the Cleveland area. 


3D) AEP and PJM Begin Arranging a TLB for 
Star-South Canton: 15:35 EOT 


Because its alarm system was not 
working, FE was not aware of the 
Harding-Chamberlin or Hanna- 
Juniper line trips. However, once 
MISO manually updated the state 
estimator model for the Stuait-Atlanta 345-kV line 
outage, the software successfully completed a 
state estimation and contingency analysis at 15:41 



EOT. But this left a 36 minute period, from 15:05 
EOT to 15:41 EDT, during which MISO did not 
recognize the consequences of the Hanna-Juniper 
loss, and FE operators knew neither of the line’s 
loss nor its consequences, PJM and AEP recog- 
nized the overload on Star-South Canton, but had 
not expected it because their earlier contingency 
analysis did not examine enough lines within the 
FE system to foresee this result of the Hanna- 
Juniper contingency on lop of the Harding- 
Chamberlin outage. 

After AEP recognized the Star- 
South Canton overload, at 15:35 
EDT AEP asked PJM to begin 
developing a 350-MW TLR to mit- 
igate it. The TLR was to relieve 
the actual overload above normal rating then 
occurring on Star-South Canton, and prevent an 
overload above emergency rating on that line if the 
Sammis-Star line were to fail. But when they 
began working on the TLR, neither AEP nor PJM 
realized that the Hanna-Juniper 345-kV line had 




<> U.S.-Canada Power System Outage Task Force < N Causes of the August 14th Blackout ❖ 



253 


already tripped at 15:32 EDT, further degrading 
system conditions. Since the great majority of 
TLRs are for cuts of 25 to 50 MW, a 350-MW TLR 
request was highly unusual and operators were 
attempting to confirm why so much relief was 
suddenly required before implementing the 
requested TLR. Less than ten minutes elapsed 
between the loss of Hanna-Juniper, the overload 
above the normal limits of Star-South Canton, and 
the Star-South Canton trip and lock-out. 

The primary tool MISO uses for 
assessing reliability on key 
fiowgates (specified groupings of 
transmission lines or equipment 
that sometimes have less transfer 
capability than desired) is the flowgate monitoring 
tool. After the Harding-Chamberlin 345-kV line 
outage at 15:05 EDT, the flowgate monitoring tool 
produced incorrect (obsolete) results, because the 
outage was not reflected in the model. As a result, 
the tool assumed that Harding-Chamberlin was 
still available and did not predict an overload for 
loss of the Hanna-Juniper 345-kV line. When 
Hanna-Juniper tripped at 15:32 EDT, the resulting 
overload was detected by MISO’s SCADA and set 
off alarms to MISO’s system operators, who then 
phoned FE about it. 27 Because both MISO’s state 
estimator, which was still in a developmental 
state, and its flowgate monitoring tool were not 
working properly, MISO’s ability to recognize FE’s 
evolving contingency situation was impaired. 

3F) Loss of the Star-South Canton 345-kV Line: 
15:41 EDT 

The Star-South Canton-line (Figure 4.11) crosses 
the boundary between FE and AEP, and the line is 
jointly owned— each company owns the portion 
of the line within its respective territory and man- 
ages the right-of-way there. The Star-South Can- 
ton line tripped and reclosed three times on the 
afternoon of August 14, first at 14:27:15 EDT 
(reclosing at both ends), then at 15:38:48 EDT, and 
at 15:41:35 EDT it tripped and locked out at the 
Star substation. A short-circuit to ground occurred 
in each case. This line failed with power flow at 
93.2% of its emergency rating. 

The investigation field team 
i ns P ectec * the right of way in the 
location indicated by the relay 
I digital fault recorders, in the FE 
portion of the line. They found 
debris from trees and vegetation that had been 
felled. At this location the conductor height 
was 44 feel 9 inches. The identifiable tree remains 


measured 30 feet in height, although the team 
could not verify the location of the stump, nor find 
all sections of the tree. A nearby cluster of trees 
showed significant fault damage, including 
charred limbs and de-barking from fault current. 
Further, topsoil in the area of the tree trunk was 
disturbed, discolored mid broken up, a common 
indication of a higher magnitude fault or multiple 
faults. Analysis of another stump showed that a 
fourteen year-old tree had recently been removed 
from the middle of the right-of-way. 28 

After the Star-South Canton line was lost, flows 
increased greatly on the 138-kV system toward 
Cleveland and area voltage levels began to degrade 
on the 138-kV and 69-kV system. At the same 
time, power flows increased on the Sammis-Star 
345-kV line due to the 138-kV line trips — the only 
remaining paths into Cleveland from the south, 

FE’s operators were not aware that 
the system was operating outside 
first contingency limits after the 
Harding-Chamberlin trip (for the 
possible loss of Hanna-Juniper), 
because they did not conduct a contingency analy- 
sis. 29 The investigation team has not determined 
whether the system status information used by 
FE’s state estimator and contingency analysis 
model was being accurately updated. 

System impacts of the 345-kV failures. The inves- 
tigation modeling team examined the impact of 
the loss of the Harding-Chamberlin, Hanna- 
Juniper and Star-South Canton 345-kV lines. After 
conducting a variety of scenario analyses, they 
concluded that had either Hanna-Juniper or Har- 
ding-Chamberlin been restored and remained in- 
service, the Star-South Canton line might not have 
tripped and locked out at 15:42 EDT. 


Figure 4.11. Star-South Canton 345-kV Line 
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According to extensive investigation team model- 
ing, there were no contingency limit violations as 
of 15:05 EDT prior to the loss of the Chamberlin- 
Harding 345-kV line. Figure 4.12 shows the 
line loadings estimated by investigation team 
modeling as the 345-kV lines in northeast Ohio 
began to trip. Showing line loadings on the 345-kV 
lines as a percent of normal rating, it tracks how 
the loading on each line increased as each subse- 
quent 345-kV and 138-kV line tripped out of ser- 
vice between 15:05 EDT (Harding-Chamberlin, 
the first line above to stair-step down) and 16:06 
EDT (Dale-West Canton). As the graph shows, 
none of the 345- or 138-kV lines exceeded their 
normal ratings until after the combined trips of 
Harding-Chamberlin and Hanna-Juniper. But im- 
mediately after the second line was lost, Star- 
South Canton’s loading jumped from an estimated 
82% of normal to 120% of normal (which was still 
below its emergency rating) and remained at the 
120% level for 10 minutes before tripping out. To 
the right, the graph shows the effects of the 138-kV 
line failures (discussed in the next phase) upon 
the two remaining 345-kV lines— i.e., Sammis- 
Star’s loading increased steadily above 100% with 
each succeeding 138-kV line lost. 

Following the loss of the Harding-Chamberlin 
345-kV line at 15:05 EDT, contingency limit viola- 
tions existed for: 

♦ The Star-Juniper 345-kV line, whose loadings 
would exceed emergency limits if the Hanna- 
Juniper 345-kV line were lost; and 

♦ The Hanna-Juniper and Harding-Juniper 
345-kV lines, whose loadings would exceed 
emergency limits if the Perry generation unit 
(1,255 MW) were lost. 


Figure 4.12. Cumulative Effects of Sequential 
Outages on Remaining 345-kV Lines 



Operationally, once FE's system entered an N-l 
contingency violation state, any facility loss 
beyond that pushed them farther into violation 
and into a more unreliable state. After loss of the 
Harding-Chamberlin line, to avoid violating NERC 
criteria, FE needed to reduce loading on these 
three lines within 30 minutes such that no single 
contingency would violate an emergency limit; 
that is, to restore the system to a reliable operating 
mode. 

Phone Calls into the FE Control Boom 

■ Beginning no earlier than 14:14 
EDT when their EMS alarms 
failed, and until at least 15:42 
EDT when they began to recog- 
nize their situation, FE operators 
did not understand how much of their system was 
being lost, and did not realize the degree to which 
their perception of their system was in error ver- 
sus true system conditions, despite receiving 
clues via phone calls from AEP, PJM and MISO, 
and customers. The FE operators were not aware 
of line outages that occurred after the trip of 
Eastlake 5 at 13:31 EDT until approximately 15:45 
EDT, although they were beginning to get external 
input describing aspects of the system’s weaken- 
ing condition. Since FE’s operators were not aware 
and did not recognize events as they were occur- 
ring, they took no actions to return the system to a 
reliable state. 

A brief description follows of some of the calls FE 
operators received concerning system problems 
and their failure to recognize that the problem was 
on their system. For ease of presentation, this set 
of calls extends past the time of the 345-kV line 
trips into the time covered in the next phase, when 
the 138-kV system collapsed. 

Following the first trip of the Star-South Canton 
345-kV line at 14:27 EDT, AEP called FE at 14:32 
EDT to discuss the trip and reclose of the line. AEP 
was aware of breaker operations at their end 
(South Canton) and asked about operations at FE’s 
Star end. FE indicated they had seen nothing at 
their end of the line but AEP reiterated that the trip 
occurred at 14:27 EDT and that the South Canton 
breakers had reclosed successfully. 30 There was 
an internal FE conversation about the AEP call at 
14:51 EDT, expressing concern that they had not 
seen any indication of an operation, but lacking 
evidence within their control room, the FE opera- 
tors did not pursue the issue. 

At 15:19 EDT, AEP called FE back to confirm that 
the Star-South Canton trip had occurred and that 
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AEP had a confirmed relay operation from the site. 
FE’s operator restated that because they had 
received no trouble or alarms, they saw no prob- 
lem. An AEP technician at the South Canton sub- 
station verified the trip. At 15:20 EDT, AEP 
decided to treat the South Canton digital fault 
recorder and relay target information as a “fluke,” 
and checked the carrier relays to determine what 
the problem might be. 31 

At 15:35 EDT the FE control center received a call 
from the Mansfield 2 plant operator concerned 
about generator fault recorder triggers and excita- 
tion voltage spikes with an alarm for over- 
excitation, and a dispatcher called reporting a 
“bump" on their system. Soon after this call, FE's 
Reading, Pennsylvania control center called 
reporting that fault recorders in the Erie west and 
south areas had activated, wondering if something 
had happened in the Ashtabula-Peny area. The 
Perry nuclear plant operator called to report a 
“spike” on the unit’s main transformer. When he 
went to look at the metering it was “still bouncing 
around pretty good. I’ve got it relay tripped up 
here ... so I know something ain’t right.” 32 

Beginning at this time, the FE operators began to 
think that something was wrong, but did not rec- 
ognize that it was on their system. “It's got to be in 
distribution, or something like that, or somebody 
else’s problem ... but I’m not showing any- 
thing.” 33 Unlike many other transmission grid 
control rooms, FE's control center does not have a 
map board (which shows schematically all major 
fines mid plants in the control area on the wall in 
front of the operators), which might have shown 
“the location of significant line and facility outages 
within the control area. 

At 15:36 EDT, MISO contacted FE regarding the 
post-contingency overload on Star-Juniper for the 
loss of the Hanna-Juniper 345-kV line. 34 

At 15:42 EDT, FE’s western transmission operator 
informed FE's IT staff that the EMS system func- 
tionality was compromised. “Nothing seems to be 
updating on the computers.... We've had people 
calling and reporting trips and nothing seems to be 
updating in the event summary. .. I think we’ve got 
something seriously sick.” This is the first evi- 
dence that a member of FE's control room staff rec- 
ognized any aspect of their degraded EMS system. 
There is no indication that he informed any of the 
other operators at this moment. However, FE’s IT 
staff discussed the subsequent EMS alarm correc- 
tive action with some control room staff shortly 
thereafter. 


Also at 15:42 EDT, the Perry plant operator called 
back with more evidence of problems. “I’m still 
getting a lot of voltage spikes and swings on the 
generator.... I don’t know how much longer we’re 
going to survive.” 35 

At 15:45 EDT, the tree trimming crew reported 
that they had witnessed a tree-caused fault on the 
Eastlake-Juniper 345-kV line; however, the actual 
fault was on the Hanna-Juniper 345-kV fine in the 
same vicinity. This information added to the con- 
fusion in the FE control room, because the opera- 
tor had indication of flow on the Eastlake-Juniper 
fine. 36 

After the Star-South Canton 345-kV fine tripped a 
third time and locked out at 15:42 EDT, AEP called 
FE at 15: 45 EDT to discuss and inform them that 
they had additional lines that showed overload. 
FE recognized then that the Star breakers had trip- 
ped and remained open. 37 

At 15:46 EDT the Perry plant operator called the 
FE control room a third time to say that the unit 
was close to tripping off: “It’s not looking good.... 
We ain’t going to be here much longer and you’re 
going to have a bigger problem.” 38 

At 15:48 EDT, an FE transmission operator sent 
staff to man the Star substation, and then at 15:50 
EDT, requested staffing at the regions, beginning 
with Beaver, then East Springfield. 39 

At 15:48 EDT, PJM called MISO to report the 
Star-South Canton trip, but the two reliability 
coordinators’ measures of the resulting fine flows 
on FE’s Sammis-Star 345-kV line did not match, 
causing them to wonder whether the Star-South 
Canton 345-kV fine had returned to service. 40 
At 15:56 EDT, because PJM was still concerned 
about the impact of the Star-South Canton trip, 
PJM called FE to report that Star-South Canton 
had tripped and that PJM thought FE’s 
Sammis-Star line was in actual emergency limit 
overload. FE could not confirm this overload. FE 
informed PJM that Hanna-Juniper was also out 
service. FE believed that the problems existed 
beyond their system. “AEP must have lost some 
major stuff.” 41 
Emergency Action 

For FirstEnergy, as with many utilities, emergency 
awareness is often focused on energy shortages. 
Utilities have plans to reduce loads under these 
circumstances to increasingly greater degrees. 
Tools include calling for contracted customer load 
reductions, then public appeals, voltage reduc- 
tions, and finally shedding system load by cutting 
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off interruptible and firm customers. FE has a plan 
for this that is updated yearly. While they can trip 
loads quickly where there is SCADA control of 
load breakers (although FE has few of these), from 
an energy point of view, the intent is to be able to 
regularly rotate what loads are not being served, 
which requires calling personnel out to switch the 
various groupings in and out. This event was not, 
however, a capacity or energy emergency or sys- 
tem instability, but an emergency due to transmis- 
sion line overloads. 

To handle an emergency effectively a dispatcher 
must first identify the emergency situation and 
then determine effective action. AEP identified 
potential contingency overloads at 15:36 EDT and 
called PJM even as Star-South Canton, one of the 
AEP/FE lines they were discussing, tripped and 
pushed FE's Sammis-Star 345-kV line to its emer- 
gency rating. Since that event was the opposite of 
the focus of their discussion about a TLR for a pos- 
sible loss of Sammis-Star that would overload 
Star-South Canton, they recognized that a serious 
problem had arisen on the system for which they 
did not have a ready solution. 42 Later, around 
15:50 EDT, their conversation reflected emer- 
gency conditions (138-kV lines were tripping and 
several other lines overloaded) but they still found 
no practical way to mitigate these overloads across 
utility and reliability coordinator boundaries. 

At the control area level, FE remained unaware of 
the precarious condition their system was in, with 
key lines out of service, degrading voltages, and 
severe overloads on their remaining lines. 43 Tran- 
scripts show that FE operators were aware of fall- 
ing voltages and customer problems after loss of 
the Hanna-Juniper 345-kV line (at 15:32 EDT). 
They called out personnel to staff substations 
because they did not think they could see them 
with their data gathering tools. They were also 
talking to customers. But there is no indication 
that FE’s operators clearly identified their situa- 
tion as a possible emergency until around 15:45 
EDT when the shift supervisor informed his man- 
ager that it looked as if they were losing the sys- 
tem: even then, although FE had grasped that its 
system was in trouble, it never officially declared 
that it was an emergency condition and that emer- 
gency or extraordinary action was needed. 

FE’s internal control room procedures and proto- 
cols did not prepare them adequately to identify 
and react to the August 14 emergency. Through- 
out the afternoon of August 14 there were many 
clues that FE had lost both its critical monitoring 
alarm functionality and that its transmission 


system’s reliability was becoming progressively 
more compromised. However, FE did not fully 
piece these clues together until after it had already 
lost critical elements of its transmission system 
and only minutes before subsequent trippings 
triggered the cascade phase of the blackout. The 
clues to a compromised EMS alarm system and 
transmission system came from a number of 
reports from various parties external to the FE 
transmission control room. Calls from FE custom- 
ers, generators, AEP, MISO and PJM came into the 
FE control room. In spite of these clues, because of 
a number of related factors, FE failed to identify 
the emergency that it faced. 

The most critical factor delaying the assessment 
and synthesis of the clues was a lack of informa- 
tion sharing between the FE system operators. In 
interviews with the FE operators and analysis of 
phone transcripts, it is evident that rarely were 
any of the critical clues shared with fellow opera- 
tors. This lack of information sharing can be 
attributed to: 


1. Physical separation of operators (the reliability 
operator responsible for voltage schedules is 
across the hall from the transmission 
operators). 

2. The lack of a shared electronic log (visible to 
all), as compared to FE’s practice of separate 
hand -written logs. 44 

3. Lack of systematic procedures to brief incoming 
staff at shift change times. 

4. Infrequent training of operators in emergency 
scenarios, identification and resolution of bad 
data, and the importance of sharing key infor- 
mation throughout the control room, 



FE has specific written proce- 
dures and plans for dealing with 
resource deficiencies, voltage 
depressions, and overloads, and 
these include instructions to 
adjust generators and trip firm loads. After the loss 
of the Star-South Canton line, voltages were below 
limits, and there were severe line overloads. But 
FE did not follow any of these procedures on 
August 14, because FE did not know for most of 
that time that its system might need such 
treatment. 


MISO was hindered because it 
lacked clear visibility, responsi- 
bility, authority, and ability to 
take the actions needed in this cir- 
cumstance. MISO had interpre- 
tive and operational tools and a large amount of 
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Figure 4.13. Timeline Phase 4 
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system data, but had a limited view of FE’s system. 
In MISO’s function as FE’s reliability coordinator, 
its primary task was to initiate and implement 
TLRs, recognize and solve congestion problems in 
less dramatic reliability circumstances with lon- 
ger solution time periods than those which existed 
on August 14. 

What training did the operators and reliability 
coordinators have for recognizing and responding 
to emergencies? FE relied upon on-the-job experi- 
ence as training for its operators in handling the 
routine business of a normal day but had never 
experienced a major disturbance and had no simu- 
lator training or formal preparation for recogniz- 
ing and responding to emergencies. Although all 
affected FE and MISO operators were NERC certi- 
fied, neither group had significant-training, docu- 
mentation, or actual experience for how to handle 
an emergency of this type and magnitude. 

Throughout August 14, most major elements of 
FE’s EMS were working properly. The system was 
automatically transferring accurate real-time 
information about FE’s system conditions to com- 
puters at AEP, MISO, and PJM. FE's operator did 
not believe the transmission line failures reported 
by AEP and MISO were real until 15:42 EDT, after 
FE conversations with tire AEP and MISO control 
rooms and calls from FE IT staff to report the fail- 
ure of their alarms. At that point in time, FE opera- 
tors began to think that their system might be in 
jeopardy — but they did not act to restore any of the 
lost transmission lines, clearly alert their reliabil- 
ity coordinator or neighbors about their situation, 
or take other possible remedial measures (such as 
load-shedding) to stabilize their system. 


Phase 4: 

138-kV Transmission System 
Collapse in Northern Ohio: 
15:39 to 16:08 EDT 

Overview of This Phase 

As each of FE’s 345-kV lines in the Cleveland area 
tripped out, it increased loading and decreased 
voltage on the underlying 138-kV system serving 
Cleveland and Akron, pushing those lines into 
overload. Starting at 15:39 EDT, the first of an 
eventual sixteen 138-kV lines began to fail. Figure 
4.14 shows how actual voltages declined at key 
138-kV buses as the 345- and 138-kV lines were 
lost. As these lines failed, the voltage drops caused 
a number of large industrial customers with volt- 
age-sensitive equipment to go off-line automati- 
cally to protect their operations. As the 138-kV 
lines opened, they blacked out customers in 

Figure 4.14. Voltages on FirstEnergy’s 138-kV 
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Akron and the areas west and south of the city, 

ultimately dropping about 600 MW of load. 

Key Phase 4 Events 

Between 15:39 EDT and 15:58:47 EDT seven 

138-kV lines tripped: 

4A) 15:39:17 EDT: Pleasant Valley-West Akron 
138-kV line tripped and reclosed at both ends. 
15:42:05 EDT: Pleasant Valley-West Akron 
138-kV West line tripped and reclosed. 
15:44:40 EDT: Pleasant Valley-West Akron 
138-kV West line tripped and locked out. 

4B) 15:42:49 EDT: Canton Central-Cloverdale 
138-kV line tripped and reclosed. 

15:45:39 EDT: Canton Central-Cloverdale 
138-kV line tripped and locked out. 

4C) 15:42:53 EDT: Cloverdale-Torrey 138-kV line 
tripped. 

4D) 15:44:12 EDT: East Lima-New Liberty 138-kV 
line tripped. 

4E) 15:44:32 EDT: Babb-West Akron 138-kV line 
and locked out. 

4F) 15:51:41 EDT: East Lima-N. Findlay 138-kV 
line tripped and reclosed at East Lima end 
only. 

4G) 15:58:47 EDT: Chamberlin-West Akron 138- 
kV line tripped. 

Note: 15:51:41 EDT: Fostoria Central-N. 
Findlay 138-kV line tripped and reclosed, but 
never locked out_ 

At 15:59:00 EDT, the loss of the West Akron bus 

caused another five 138-kV lines to trip: 

4H) 15:59:00 EDT: West Akron 138-kV bus trip- 
ped, and cleared bus section circuit breakers 
at West Akron 138 kV. 

41) 15:59:00 EDT: West Akron-Aetna 138-kV line 
opened. 

4J) 15:59:00 EDT: Barberton 138-kV line opened 
at West Akron end only. West Akron-B18 
138-kV tie breaker opened, affecting West 
Akron 138/12-kV transformers #3, 4 and 5 fed 
from Barberton. 

4K) 15:59:00 EDT: West Akron-Granger-Stoney- 
Brunswick-West Medina opened. 

4L) 15:59:00 EDT: West Akron-Pleasant Valley 
138-kV East line (Q-22) opened. 


4M) 15:59:00 EDT: West Akron-Rosemont-Pine- 
Wadsworth 138-kV line opened. 

From 16:00 EDT to 16:08:59 EDT, four 138-kV 
lines tripped, and the Sammis-Star 345-kV line 
tripped on overload: 

4N) 16:05:55 EDT: Dale-West Canton 138-kV line 
tripped at both ends, reclosed at West Canton 
only 

40) 16:05:57 EDT: Sammis-Star 345-kV line 
tripped 

4P) 16:06:02 EDT: Star-Urban 138-kV line tripped 
4Q) 16:06:09 EDT: Richland-Ridgeville-Napo- 
leon-Stryker 138-kV line tripped and locked 
out at all terminals 

4R) 16:08:58 EDT: Ohio Central-Wooster 138-kV 
line tripped 

Note: 16:08:55 EDT: East Wooster-South Can- 
ton 138-kV line tripped, but successful auto- 
matic reclosing restored this line. 

4A-G) Pleasant Valley to Chamberlin-West 
Akron Line Outages 

From 15:39 EDT to 15:58:47 EDT, seven 138-kV 
lines in northern Ohio tripped and locked out. At 
15:45:41 EDT, Canton Central-Tidd 345-kV line 
tripped and reclosed at 15:46:29 EDT because 
Canton Central 345/138-kV CB “Al” operated 
multiple times, causing a low air pressure problem 
that inhibited circuit breaker tripping. This event 
forced the Canton Central 345/138-kV transform- 
ers to disconnect and remain out of service, fur- 
ther weakening the Canton- Akron area 138-kV 
transmission system. At 15:58:47 EDT the 
Chamberlin-West Akron l38-kV line tripped. 

4H-M) West Akron Transformer Circuit 
Breaker Failure and Line Outages 

At 15:59 EDT FE’s West Akron 138-kV bus tripped 
due to a circuit breaker failure on West Akron 
transformer #1. This caused the five remaining 
138-kV lines connected to the West Akron 
substation to open. The West Akron 138/12-kV 
transformers remained connected to the Barber- 
ton-West Akron 138-kV line, but power flow to 
West Akron 138/69-kV transformer #1 was 
interrupted. 

4N-0) Dale-West Canton 138-kV and 
Sammis-Star 345-kV Lines Tripped 
After the Cloverdale-Torrey line failed at 15:42 
EDT, Dale-West Canton was the most heavily 
loaded line on FE's system. It held on, although 
heavily overloaded to 160 and 180% of normal 
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ratings, until tripping at 16:05:55 EDT. The loss of 
this line had a significant effect on the area, and 
voltages dropped significantly. More power 
shifted back to the remaining 345-kV network, 
pushing Sammis-Star’s loading above 120% of rat- 
ing. Two seconds later, at 16:05:57 EDT, Sammis- 
Star tripped out. Unlike the previous three 345-kV 
lines, which tripped on short circuits to ground 
due to tree contacts, Sammis-Star tripped because 
its protective relays saw low apparent impedance 
(depressed voltage divided by abnormally high 
line current) — i.e., the relay reacted as if the high 
flow was due to a short circuit. Although three 
more 138-kV lines dropped quickly in Ohio fol- 
lowing the Sammis-Star trip, loss of the Sammis- 
Star line marked the turning point at which sys- 
tem problems in northeast Ohio initiated a cascad- 
ing blackout across the northeast United States 
and Ontario. 45 
Losing the 138-kV System 

The tripping of 138-kV transmission lines that 
began at 15:39 EDT occurred because the loss 
of the combination of the Harding-Chamberlin, 
Hanna-Juniper and Star-South Canton 345-kV 
lines overloaded the 138-kV system with electric- 
ity flowing north toward the Akron and Cleveland 
loads. Modeling indicates that the return of either 
the Hanna-Juniper or Chamberlin-Harding 345-kV 
lines would have diminished, but not alleviated, 
all of the 138-kV overloads. In theory, the return of 
both lines would have restored all the 138 lines to 
within their emergency ratings. 

However, all three 345-kV lines 
had already been compromised 
due to tree contacts so it is 
unlikely that FE would have suc- 
cessfully restored either line had 
they known it had tripped out, and since 
Star-South Canton had already tripped and 
reclosed three times it is also unlikely that an 
operator knowing this would have trusted it to 
operate securely under emergency conditions. 
While generation redispatch scenarios alone 
would not have solved the overload problem, 
modeling indicates that shedding load in the 
Cleveland and Akron areas may have reduced 
most line loadings to within emergency range and 
helped stabilize the system. However, the amount 
of load shedding required grew rapidly as FE’s sys- 
tem unraveled. 

Loss of the Sammis-Star 345-kV Line 

Figure 4.15, derived from investigation team mod- 
eling, shows how the power flows shifted across 


FE’s 345- and key 138-kV northeast Ohio lines as 
the line failures progressed. All lines were 
loaded within normal limits after the Harding- 
Chamberlin lock-out, but after the Hanna-Juniper 
trip at 15:32, the Star-South Canton 345-kV line 
and three 138-kV lines jumped above normal load- 
ings. After Star-South Canton locked out at 15:41 
EDT, five 138-kV and the Sammis-Star 345-kV 
lines were overloaded and Star-South Canton was 
within its emergency rating. From that point, as 
the graph shows, each subsequent line loss 
increased loadings on other lines, some loading to 
well over 150% of normal ratings before they 
failed. The Sammis-Star 345-kV line stayed in ser- 
vice until it tripped at 16:05:57 EDT. 


Figure 4.15. Simulated Effect of Prior Outages on 
138-kV Line Loadings 



Endnotes 

1 August 14, 2003 Outage Sequence of Events, U.S./Canada 
Power Outage Task Force (September 12, 2003), httpaVwww. 
electricity.doe.gov/documen ts/l 2820031 13351_BlackoutSummary.pdf. 
2 DOE Site Visit to FE 10/8/2003: Steve Morgan. 

3 DOE Site Visit to FE, September 3, 2003, Hough interview: 
‘"When asked whether the voltages seemed unusual, he said 
that some sagging would be expected on a hot day, but on 
August 14th the voltages did seem unusually low.” Spidle 
interview: ‘The voltages for the day were not particularly 
bad.” 

4 Manual of Operations, valid as of March 3, 2003, Process 
flowcharts: Voltage Control and Reactive Support - Plant and 
System Voltage Monitoring Under Normal Conditions. 
5 14:13:18. Channel 16 - Sammis 1. 13:15:49 / Channel 16 - 
West Lorain (FE Reliability Operator (RO) says, “Thanks. 
We're starting to sag all over the system.”) / 1 3:16:44. Channel 
16 - Eastlake (talked to two operators) (RO says, “We got a 
way bigger load than we thought we would have.” And “.. .So 
we’re starting to sag all over the system.”) / 13:20:22. Channel 
16 - RO to “Berger” / 13:22:07. Channel 16 - “control room" 
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RO says, “We’re sagging all over the system. I need some 
help." / 13:23:24. Channel 16 - “Control room, Tom” / 
13:24:38. Channel 16 - “Unit 9” / 13:28:04. Channel 16 - 
“Dave” / 13:28:40. Channel 16 "Troy Control”. Also general 
note in RO Dispatch Log. 

6 Example at 13:33:40, Channel 3, FE transcripts. 

7 Investigation Team Site Visit to MISO, Walsh and Seidu 
interviews. 

®FE had and ran a state estimator every 30 minutes. This 
served as a base from which to perform contingency analyses. 
PE’s contingency analysis tool used SCADA and EMS inputs 
to identify any potential overloads that could result from vari- 
ous line or equipment outages. FE indicated that it has experi- 
enced problems with the automatic contingency analysis 
operation since the system was installed in 1995. As a result, 
FE operators or engineers ran contingency analysis manually 
rather than automatically, and were expected to do so when 
there were questions about the state of the system. Investiga- 
tion team interviews of FE personnel indicate that the contin- 
gency analysis model was likely running but not consulted at 
any point in the afternoon of August 14. 

9 After the Stuart-Atlanta line tripped, Dayton Power & Light 
did not immediately provide an update of a change in equip- 
ment availability using a standard form that posts the status 
change in the SDX (System Data Exchange, the NERC data- 
base which maintains real-time information on grid equip- 
ment status), which relays that notice to reliability 
coordinators and control areas. After its state estimator failed 
to solve properly, MISO checked the SDX to make sure that 
they had properly identified all available equipment and out- 
ages, but found no posting there regarding Stuart-Atianta’s 
outage. 

10 Investigation team field visit, interviews with FE personnel 
on October 8-9, 2003. 

11 DOE Site Visit to First Energy, September 3, 2003, Inter- 
view with David M. Elliott. 

12 FE Report, "Investigation of FirstEnergy's Energy Manage- 
ment System Status on August 14, 2003”, Bullet 1, Section 
4.2.11. 

13 Investigation team interviews with FE, October 8-9, 2003. 

- 14 DOE Site Visifat FE, October 8-9, 2003; investigation team 

was advised that FE had discovered this effect during 
post-event investigation and testing of the EMS. FE*s report 
“Investigation of FirstEnergy’s Energy Management System 
Status on August 14, 2003” also indicates that this finding 
was “verified using the strip charts from 8-14-03” (page 23), 
not that the investigation of this item was instigated by opera- 
tor reports of such a failure. 

15 There is a conversation between a Phil and a Tom that 
speaks of “flatlming” 15:01:33. Channel 15. There is no men- 
tion of AGC or generation control in the DOE Site Visit inter- 
views with the reliability coordinator. 

16 DOE Site Visit to FE, October 8-9, 2003, Sanicky Interview: 
"From his experience, it is not unusual for alarms to fail. 
Often times, they may be slow to update or they may die com- 
pletely. From his experience as a real-time operator, the fact 
that the alarms failed did not surprise him.” Also from same 
document, Mike McDonald interview "FE has previously bad 
(servers] down at the same time. The big issue for them was 
that they were not receiving new alarms.” 

17 A “cold” reboot of the XA21 system is one in which all 
nodes (computers, consoles, etc.) of the system are shut down 
and then restarted. Alternatively, a given XA21 node can be 


“warm” rebooted wherein only that node is shut down and 
restarted, or restarted from a shutdown stale. A cold reboot 
will take significantly longer to perform than a warm one. 
Also during a cold reboot much more of the system is unavail- 
able for use by the control room operators for visibility or con- 
trol over the power system. Warm reboots are not uncommon, 
whereas cold reboots are rare. All reboots undertaken by FE’s 
IT EMSS support personnel on August 14 were warm reboots, 
18 The cold reboot was done in the early morning of 1 5 August 
and corrected the alarm problem as hoped. 

19 Example at 14:19, Channel 14, FE transcripts. 

20 Example at 14:25, Channel 8, FE transcripts. 

25 Example at 14:32, Channel 15, FE transcripts. 

22 Investigation team transcript, meeting on September 9, 
2003, comments by Mr. Steve Morgan, Vice President Electric 
Operations: 

Mr. Morgan: The sustained outage history for these lines, 
2001, 2002, 2003, up until the event, Chamberlin-Harding 
had zero operations for those two-and-a-half years. And 
Hanna-Juniper had six operations in 2001, ranging from four 
minutes to maximum of 34 minutes. Two were unknown, one 
was lightning, one was a relay failure, and two were really 
relay scheme mis -operations. They're category other. And 
typically, that — I don't know what this is particular to opera- 
tions, that typically occurs when there is a mis -ope ration. 
Star-South Canton had no operations in that same period of 
time, two-and-a-half years. No sustained outages. And 
Sammis-Star, the line we haven’t talked about, also no sus- 
tained outages during that two-and-a-half year period. 

So is it normal? No. But 345 lines do operate, so it's not 
unknown. 

23 “Interim Report, Utility Vegetation Management,” 
U.S.-Canada Joint Outage investigation Task Force, Vegeta- 
tion Management Program Review, October 2003, page 7. 

24 Investigation team October 2, 2003, fact-finding meeting, 

' Steve Morgan statement. 

25 "FE MISO Findings,” page 11. 

26 FE was conducting right-of-way vegetation maintenance on 
a 5-year cycle, and the tree crew at Hanna-Juniper was three 
spans away, clearingjvegetation near the line, when the con- 
tact occurred on August 14. Investigation team 9/9/03 meet- 
ing transcript, and investigation field team discussion with 
the tree-trimming crew foreman. 

27 Based on “FE MISO Findings” document, page 11 . 

28 "Interim Report, Utility Vegetation Management," 
US-Canada Joint Outage Task Force, Vegetation Management 
Program Review, October 2003, page 6. 

29 Investigation team September 9, 2003 meeting transcripts, 
Mr. Steve Morgan, First Energy Vice President, Electric Sys- 
tem Operations: 

Mr. Benjamin: Steve, just to make sure that I’m understand- 
ing it coiTectly, you had indicated that once after 
Hanna-Juniper relayed out, there wasn’t really a problem 
with voltage on the system until Star-S. Canton operated. But 
were the system operators aware that when Hanna-Juniper 
was out, that if Star-S. Canton did trip, they would be outside 
of operating limits? 

Mr. Morgan: I think the answer to that question would have 
required a contingency analysis to be done probably on 
demand for that operation. It doesn't appear to me that a con- 
tingency analysis, and certainly not a demand contingency 
analysis, could have been run in that period of time. Other 
than experience. I don’t know that they would have been able 
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to answer that question. And what I know of the record right 
now is that it doesn’t appear that they ran contingency analy- 
sis on demand. 

Mr. Benjamin: Could they have done that? 

Mr. Morgan: Yeah, presumably they could have. 

Mr. Benjamin: You have all the tools to do that? 

Mr. Morgan: They have all the tools and all the information is 
there. And if the State Estimator is successful in solving, and 
all the data is updated, yeah, they could have. I would say in 
addition to those tools, they also have access to the planning 
load flow model that can actually run the same — full load of 
the model if they want to. 

30 Example synchronized at 14:32 (from 13:32) #18 041 
TDC-E2 283.wav, AEP transcripts. 

31 Example synchronized at 14:19 #2 020 TDC-El 266.wav, 
AEP transcripts. 

32 Example at 15:36 Channel 8, FE transcripts. 

33 Example at 15:41:30 Channel 3, FE transcripts. 

34 Example synchronized at 15:36 (from 14:43) Channel 20, 
M3SO transcripts. 

35 Example at 15:42:49, Channel 8. FE transcripts. 
36 Example at 15:46:00, Channel 8 FE transcripts. 

37 Example at 15:45:18, Channel 4, FE transcripts. 

38 Example at 15:46:00, Channel 8 FE transcripts. 


39 Example at 15:50:15, Channel 12 FE transcripts. 

40 Example synchronized at 15:48 (from 14:55), channel 22, 
MISO transcripts. 

4I Example at 15:56:00, Channel 31, FE transcripts, 

42 AEP Transcripts CAEl 8/14/2003 14:35 240. 

43 FE Transcripts 15:45:18 on Channel 4 and 15:56:49 on 
Channel 31. 

44 The operator logs from FE’s Ohio control center indicate 
that the west desk operator knew of the alarm system failure 
at 14:14, but that the east desk operator first knew of this 
development at 15:45. These entries may have been entered 
after the times noted, however, 

45 The investigation team determined that FE was using a dif- 
ferent set of line ratings for Sammis-Star than those being 
used in the MISO and PJM reliability coordinator calcula- 
tions or by its neighbor AEP. Specifically, FE was operating 
Sammis-Star assuming that the 345-kV line was rated for 
summer normal use at 1,310 MVA, with a summer emer- 
gency limit rating of 1,310 MVA. In contrast, MISO, PJM and 
AEP were using a mean conservative rating of 950 MVA nor- 
mal and 1,076 MVA emergency for this line. The facility 
owner (in this case FE) is the entity which provides the line 
rating; when and why the ratings were changed and not com- 
municated to all concerned parties has not been determined. 
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5. The Cascade Stage of the Blackout 


Chapter 4 described how uncorrected problems in 
northern Ohio developed to a point that a cascad- 
ing blackout became inevitable. However, the 
Task Force’s investigation also sought to under- 
stand how and why the cascade spread and 
stopped as it did. As detailed below, the investiga- 
tion determined the sequence of events in the cas- 
cade, and in broad terms how it spread and how it 
stopped in each general geographic area. 1 

Why Does a Blackout Cascade? 

Major blackouts are rare, and no two blackout sce- 
narios are the same. The initiating events will 
vary, including human actions or inactions, sys- 
tem topology, and load/generation balances. Other 
factors that will vary include the distance between 
generating stations and major load centers, voltage 
profiles, and the types and settings of protective 
relays in use. 

Most wide-area blackouts start with short circuits 
(faults) on several transmission lines in short suc- 
cession — sometimes resulting from natural causes 
such as lightning or wind or, as on August 14, 
resulting from inadequate tree management in 
right-of-way areas. A fault causes a high current 
and low voltage on the line containing the fault. A 
protective relay for that line detects the high cur- 
rent and low voltage and quickly trips the circuit 
breakers to isolate that line from the rest of the 
power system. 

A cascade occurs when there is a sequential trip- 
ping of numerous transmission lines and genera- 
tors in a widening geographic area. A cascade can 
be triggered by just a few initiating events, as was 
seen on August 14. Power swings and voltage fluc- 
tuations caused by these initial events can cause 
other lines to detect high currents and low volt- 
ages that appear to be faults, even when faults do 
not actually exist on those other lines. Generators 
are tripped off during a cascade to protect them 
from severe power and voltage swings. Relay pro- 
tection systems work well to protect lines and gen- 
erators from damage and to isolate them from 
the system under normal, steady conditions. 


However, when power system operating and 
design criteria are violated as a result of several 
outages occurring at the same time, most common 
protective relays cannot distinguish between the 
currents and voltages seen in a system cascade 
from those caused by a fault. This leads to more 
and more lines and generators being tripped, wid- 
ening the blackout area. 

How Did the Cascade Evolve on 
August 14? 

At 16:05:57 Eastern Daylight Time, the trip and 
lock-out of FE's Sammis-Star 345 kV line set off a 
cascade of interruptions on the high voltage sys- 
tem, causing electrical fluctuations and facility 
trips as within seven minutes the blackout rippled 
from the Akron area across much of the northeast 
United States and Canada. By 16:13 EDT, more 
than 263 power plants (531 individual generating 
units) had been lost, and tens of millions of people 
in the United States and Canada were without 
electric power. 

Chapter 4 described the four phases that led to the 
initiation of the cascade at about 16:06 EDT. After 
16:06 EDT, the cascade evolved in three distinct 
phases: — 

♦ Phase 5. The collapse of FE’s transmission sys- 
tem induced unplanned power surges across 
the region. Shortly before the collapse, large 
electricity flows were moving across FE’s sys- 
tem from generators in the south (Tennessee, 
Kentucky, Missouri) to load centers in northern 
Ohio, eastern Michigan, and Ontario. This path- 
way in northeastern Ohio became unavailable 
with the collapse of FE’s transmission system. 
The electricity then took alternative paths to the 
load centers located along the shore of Lake 
Erie. Power surged in from western Ohio and 
Indiana on one side and from Pennsylvania 
through New York and Ontario around the 
northern side of Lake Erie. Transmission lines 
in these areas, however, were already heavily 
loaded with normal flows, and some of them 
began to trip. 
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♦ Phase 6. The northeast then separated from the 
rest of the Eastern Interconnection due to these 
additional power surges. The power surges 
resulting from the FE system failures caused 
lines in neighboring areas to see overloads that 
caused impedance relays to operate. The result 
was a wave of line trips through western Ohio 
that separated AEP from FE. Then the line trips 
progressed northward into Michigan separating 
western and eastern Michigan. 

With paths cut from the west, a massive power 
surge flowed from PJM into New York and 
Ontario in a counter-clockwise flow around 
Lake Erie to serve the load still connected in 
eastern Michigan and northern Ohio. The relays 
on the lines between PJM and New York saw 
this massive power surge as faults and tripped 
those lines. Lines in western Ontario also 
became overloaded and tripped. The entire 
northeastern United Stales and the province of 
Ontario then became a large electrical island 
separated from the rest of the Eastern Intercon- 
nection. This large island, which had been 
importing power prior to the cascade, quickly 
became unstable as there was not sufficient gen- 
eration in operation within it to meet electricity 
demand. Systems to the south and west of the 


split, such as PJM, AEP and others further away 
remained intact and were mostly unaffected by 
the outage. Once the northeast split from the 
rest of the Eastern Interconnection, the cascade 
was isolated. 

Phase 7. In the final phase, the large electrical 
island in the northeast was deficient in generation 
and unstable with large power surges and swings 
in frequency and voltage. As a result, many lines 
and generators across the disturbance area trip- 
ped, breaking the area into several electrical 
islands. Generation and load within these smaller 
islands was often unbalanced, leading to further 
tripping of lines and generating units until equi- 
librium was established in each island. Although 
much of the disturbance area was fully blacked 
out in this process, some islands were able to 
reach equilibrium without total loss of service. For 
example, most of New England was stabilized and 
generation and load restored to balance. Approxi- 
mately half of the generation and load remained 
on in western New York, which has an abundance 
of generation. By comparison, other areas with 
large load centers and insufficient generation 
nearby to meet that load collapsed into a blackout 
condition (Figure 5.1). 
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Figure 5.1. Area Affected by the Blackout 



What Stopped the August 14 Blackout 
from Cascading Further? 

The investigation concluded that one or more of 
the following likely determined where and when 
the cascade stopped spreading: 

♦ The effects of a disturbance travel over power 
lines and become dampened the further they 
are from the initial point, much like the ripple 
from a stone thrown in a pond. Thus, the volt- 
age and current swings seen by relays on lines 
farther away from the initial disturbance are not 
as severe, and at some point they are no longer 
sufficient to induce lines to trip. 

♦ Higher voltage lines and more densely net- 
worked lines, such as the 500-kV system in PJM 
and the 765-kV system in AEP, are better able to 
absorb voltage and current swings and thus 
serve as a barrier to the spreading of a cascade. 
As seen in Phase 6, the cascade progressed into 
western Ohio and then northward through 
Michigan through the areas that had the fewest 
transmission lines. Because there were fewer 
lines, each line absorbed more of the power and 
voltage surges and was more vulnerable to trip- 
ping, A similar effect was seen toward the east 
as the lines between New York and Pennsylva- 
nia, and eventually northern New Jersey trip- 
ped, The cascade of transmission line outages 
became isolated after the northeast United 
States and Ontario were completely separated 
from the rest of the Eastern Interconnection and 
no more power flows were possible into the 
northeast (except the DC ties from Quebec, 
which continued to supply power to western 
New York and New England). 

♦ Some areas, due to line trips, were isolated from 
the portion of the grid that was experiencing 
instability. Many of these areas retained 
sufficient on-line generation or the capacity to 


import power from other parts of the grid, unaf- 
fected by the surges or instability, to meet 
demand. As the cascade progressed, and more 
generators and lines tripped off to protect them- 
selves from severe damage, and some areas 
completely separated from the unstable part of 
the Eastern Interconnection. In many of these 
areas there was sufficient generation to stabilize 
the system. After the large island was formed in 
the northeast, symptoms of frequency and volt- 
age collapse became evident. In some parts of 
the large area, the system was too unstable and 
shut itself down. In other parts, there was suffi- 
cient generation, coupled with fast-acting auto- 
matic load shedding, to stabilize frequency and 
voltage. In this manned, most of New England 
remained energized. Approximately half of the 
generation and load remained on in western 
New York, aided by generation in southern 
Ontario that split and stayed with western New 
York. There were other smaller isolated pockets 
of load and generation that were able to achieve 
equilibrium and remain energized. 

Phase 5: 

345-kV Transmission System 
Cascade in Northern Ohio and 
South-Central Michigan 
Overview of TMs Phase 

This initial phase of the cascade began because 
after the loss of FE’s Sammis-Star 345-kV line and 
the underlying 138-kV system, there were no large 
transmission paths left from the south to support 
the significant amount of load in northern Ohio 
(Figure 5.2). This placed a significant load burden 


Figure 5.2. Sammis-Star 345-kV Line Trip, 
16:05:57 EOT 
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onto the transmission paths north and northwest 
into Michigan, causing a steady loss of lines and 
power plants. 

Key Events in This Phase 
5A) 16:05:57 EDT: Sammis-Star 345-kV tripped. 
5B) 16:08:59 EDT: Galion-Ohio Central-Musk- 
ingum 345-kV line tripped. 

5C) 16:09:06 EDT: East Lima-Fostoria Central 
345-kV line tripped, causing major power 
swings through New York and Ontario into 
Michigan. 

5D) 16:09:08 EDT to 16:10:27 EDT: Several power 
plants lost, totaling 937 MW. 

5 A) Sammis-Star 345-kV Tripped: 16:05:57 EDT 
Sammis-Star did not trip due to a short circuit to 
ground (as did the prior 345-kV lines that tripped). 
Sammis-Star tripped due to protective relay action 
that measured low apparent impedance (de- 
pressed voltage divided by abnormally high line 
current) (Figure 5.3). There was no fault and no 
major power swing at the time of the trip — -rather, 
high flows above the line’s emergency rating 
together with depressed voltages caused the over- 
load to appear to the protective relays as a remote 
fault on the system. In effect, the relay could no 
longer differentiate between a remote three-phase 
fault and an exceptionally high line-load condi- 
tion. Moreover, the reactive flows (VArs) on the 
line were almost ten times higher than they had 
been earlier in the day. The relay operated as it 
was designed to do. 

The Sammis-Star 345-kV line trip completely sev-_ 
ered the 345-kV path into northern Ohio from 
southeast Ohio, triggering a new, fast-paced 
sequence of 345-kV transmission line trips in 
which each line trip placed a greater flow burden 


Figure 5.3. Sammis-Star 345-kV Line Trips 
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on those lines remaining in service. These line 
outages left only three paths for power to flow into 
northern Ohio: (1) from northwest Pennsylvania 
to northern Ohio around the south shore of Lake 
Erie, (2) from southern Ohio, and (3) from eastern 
Michigan and Ontario. The line interruptions sub- 
stantially weakened northeast Ohio as a source of 
power to eastern Michigan, making the Detroit 
area more reliant on 345-kV lines west and north- 
west of Detroit, and from northwestern Ohio to 
eastern Michigan. 

Transmission Lines into Northwestern Ohio 
Tripped, and Generation Tripped in South 
Central Michigan and Northern Ohio: 16:08:59 
EDT to 16:10:27 EDT 

5B) Galion-Ohio Central-Muskingum 345-kV line 
tripped: 16:08:59 EDT 

5C) East Lima-Fostoria Central 345-kV line 
tripped, causing a large power swing from 
Pennsylvania and New York through Ontario 
to Michigan: 16:09:05 EDT 

The tripping of the Galion-Ohio Central-Mus- 
kingum and East Lima-Fostoria Central 345-kV 
transmission lines removed the transmission 
paths from southern and western Ohio into north- 
ern Ohio and eastern Michigan. Northern Ohio 
was connected to eastern Michigan by only three 
345-kV transmission lines near the southwestern 
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bend of Lake Erie. Thus, the combined northern 
Ohio and eastern Michigan load centers were left 
connected to the rest of the grid only by: (1) trans- 
mission lines eastward from northeast Ohio to 
northwest Pennsylvania along the southern shore 
of Lake Erie, and (2) westward by lines west and 
northwest of Detroit, Michigan and from Michigan 
into Ontario (Figure 5.4). 

The East Lima-Fostoria Central 345-kV line trip- 
ped at 16:09:06 EDT due to high currents and low 
voltage, and the resulting large power swings 
(measuring about 400 MW when they passed 
through NYPA’s Niagara recorders) marked the 
moment when the system became unstable. This 
was the first of several inter-area power and fre- 
quency events that occurred over the next two 
minutes. It was the system’s response to the loss of 
the Ohio-Michigan transmission paths (above), 
and the stress that the still-high Cleveland, Toledo 
and Detroit loads put onto the surviving lines and 
local generators. 

In Figure 5.5, a high-speed recording of 345-kV 
flows past Niagara Falls shows the New York to 
Ontario power swing, which continued to oscil- 
late for over 10 seconds. The recording shows the 
magnitude of subsequent flows triggered by the 
trips of the Hampton-Pontiac and Thetford-Jewell 
345-kV lines in Michigan and the Perry- Ashtabula 
345-kV line linking the Cleveland area to Pennsyl- 
vania. The very low voltages on the northern Ohio 
transmission system made it very difficult for the 
generation in the Cleveland and Lake Erie area to 
maintain synchronization with the Eastern Inter- 
connection. Over the next two minutes, generators 
in this area shut down after reaching a point of no 


Figure 5.4. Ohio 345-kV Lines Trip, 16:08:59 to 
16:09:07 EDT 



recovery as the stress level across the remaining 
ties became excessive. 

Before this first major power swing on the Michi- 
gan/Ontario interface, power flows in the NPCC 
Region (Ontario and the Maritimes, New England, 
New York, and the mid-Atlantic portion of PJM) 
were typical for the summer period, and well 
within acceptable limits. Transmission and gener- 
ation facilities were then in a secure state across 
the NPCC. 

5D) Multiple Power Plants Tripped, Totaling 
937 MW: 1 6:09:08 to 1 6:1 0:27 EDT 

Michigan Cogeneration Venture plant reduc- 
tion of 300 MW (from 1,263 MW to 963 MW) 

, Kinder Morgan units 1 and 2 trip (200 MW total) 
Avon Lake 7 unit trips (82 MW) 

Berger 3, 4, and 5 units trip (355 MW total) 

The Midland Cogeneration Venture (MCV) plant 
is in central Michigan. Kinder Morgan is in 
south-central Michigan. The large power reversal 
caused frequency and voltage fluctuations at the 
plants. Their automatic control systems 
responded to these transients by trying to adjust 
output to raise voltage or respond to the frequency 
changes, but subsequently tripped off-line. The 
Avon Lake and Burger units, in or near Cleveland, 
likely tripped off due to the low voltages prevail- 
ing in the Cleveland area and 138-kV line trips 
near Burger 138-kV substation (northern Ohio) 
(Figure 5.6). 

Power flows into Michigan from Indiana in- 
creased to serve loads in eastern Michigan and 
northern Ohio (still connected to the grid through 
northwest Ohio and Michigan) and voltages 


Figure 5.5. New York-Ontario Line Flows at Niagara 
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dropped from the imbalance between high 
loads and limited transmission and generation 
capability. 

Phase 6: The Full Cascade 

Between 16:10:36 EDT and 16:13 EDT, thousands 
of events occurred on the grid, driven by physics 
and automatic equipment operations. When it was 
over, much of the northeast United States and the 
Canadian province of Ontario was in the dark. 

Key Phase 6 Events 
Transmission Lines Disconnected Across 
Michigan and Northern Ohio, Generation Shut 
Donna in Central Michigan and Northern Ohio, 
and Northern Ohio Separated from 
Pennsylvania: 16:10:36 EDT to 16:10:39 EDT 

6A) Transmission and more generation tripped 
within Michigan: 16:10:36 EDT to 16:10:37 
EDT: 

Argenia-Baltlecreek 345-kV line tripped 
Battlecreek-Oneida 345-kV line tripped 
Argenta-Tompkins 345-kV line tripped 
Sumpter Units 1, 2, 3, and 4 units tripped 
(300 MW near Detroit) 

MCV Plant output dropped from 944 MW to 
109 MW. 

Together, the above line outages interrupted the 
east-to-west transmission paths into the Detroit 
area from south-central Michigan. The Sumpter 
generation units tripped in response to 
under-voltage on the system. Michigan lines 
northwest of Detroit then began to trip, as noted 
below (Figure 5.7). 


Figure 5.6. Michigan and Ohio Power Plants Trip 



6B) More Michigan lines tripped: 16:10:37 EDT to 
16:10:38 EDT 

Hampton-Pontiac 345-kV line tripped 
Thetford-JeweU 345-kV line tripped 

These 345-kV lines connect Detroit to the north. 
When they tripped out of service, it left the loads 
in Detroit, Toledo, Cleveland, and their surround- 
ing areas served only by local generation and the 
lines connecting Detroit east to Ontario and Cleve- 
land east to northeast Pennsylvania. 

6C) Cleveland separated from Pennsylvania, 
flows reversed and a huge power surge 
flowed counter-clockwise around Lake Erie: 
16:10:38.6 EDT 

Perry -Ashtabula-Erie West 345-kV line trip- 
ped: 16:10:38.6 EDT 

Large power surge to serve loads in eastern 
Michigan and northern Ohio swept across 
Pennsylvania, New Jersey, and New York 
through Ontario into Michigan: 16:10:38.6 
EDT. 

Perry- Ashtabula- West Erie was the last 345-kV 
line connecting northern Ohio to the east. This 
line’s trip separated the Ohio 345-kV transmission 
system from Pennsylvania. When it tripped, the 
load centers in eastern Michigan and northern 
Ohio remained connected to the rest of the Eastern 
Interconnection only at the interface between the 


Figure 5.7. Transmission and Generation Trips in 
Michigan, 16:10:36 to 16:10:37 EDT 
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Michigan and Ontario systems {Figure 5.8). East- 
ern Michigan and northern Ohio now had little 
internal generation left and voltage was declining. 
Between 16:10:39 EDT and 16:10:50 EDT 
under-frequency load shedding in the Cleveland 
area operated and interrupted about 1,750 MW of 
load. The frequency in the Cleveland area (by then 
separated from the Eastern Interconnection to the 
south) was also dropping rapidly and the load 
shedding was not enough to arrest the frequency 
decline. Since the electrical system always seeks 
to balance load and generation, the high loads in 
Cleveland drew power over the only major trans- 
mission path remaining — the lines from eastern 
Michigan east into Ontario. 

Before the loss of the Perry-Ashtabula-West Erie 
line, 437 MW was flowing from Michigan into 
Ontario. At 16:10:38.6 EDT, after the other trans- 
mission paths into Michigan and Ohio failed, the 
power that had been flowing over them reversed 
direction in a fraction of a second. Electricity 
began flowing toward Michigan via a giant loop 
through Pennsylvania and into New York and 
Ontario and then into Michigan via the remaining 
transmission path. Flows at Niagara Falls 345-kV 
lines measured over 800 MW, and over 3,500 MW 
at the Ontario to Michigan interface (Figure 5.9). 
This sudden large change in power flows drasti- 
cally lowered voltage and increased current levels 
on the transmission lines along the Pennsylva- 
nia-New York transmission interface. 


Figure 5.8. Michigan Lines Trip and Ohio Separates 
from Pennsylvania, 16:10:36 to 16:10:38.6 EDT 



This was a transient frequency swing, so fre- 
quency was not the same across the Eastern Inter- 
connection. As Figure 5.8 shows, tills frequency 
imbalance and the accompanying power swing 
resulted in a rapid rate of voltage decay. Flows into 
Detroit exceeded 3,500 MW and 1,500 MVAr, 
meaning that the power surge was draining both 
active and reactive power out of the northeast to 
prop up the low voltages in eastern Michigan and 
Detroit. This magnitude of reactive power draw 
caused voltages in Ontario and New York to drop. 
At the same time, local voltages in the Detroit area 
were low because there was still not enough sup- 
ply to meet load. Detroit would soon black out (as 
evidenced by the rapid power swings decaying 
after 16:10:43 EDT). 

Between 16:10:38 and 16:10:41 EDT, the power 
surge caused a sudden extraordinary increase in 
system frequency to 60.3 Hz. A series of circuits 
tripped along the border between PJM and the 
NYISO due to apparent impedance faults (short 
circuits). The surge also moved into New England 
and the Maritimes region of Canada. The combi- 
nation of the power surge and frequency rise 
caused 380 MW of pre-selected Maritimes genera- 
tion to drop off-line due to the operation of the 
New Brunswick Power "Loss of Line 3001” Special 
Protection System. Although this system was 
designed to respond to failure of the 345-kV link 
between the Maritimes and New England, it oper- 
ated in response to the effects of the power surge. 
The link remained intact during the event. 

In summary, the Perry-Ashtabula-Erie West 345- 
kV line trip at 16:1038.6 EDT was the point when 
-the Northeast entered a period of transient insta- 
bility and a loss of generator synchronism. 


Figure 5.9. Active and Reactive Power and Voltage 
from Ontario into Detroit 
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Western Pennsylvania Separated from New 
York: 16:10:39 EDT to 16:10:44 EDT 

6D) 16:10:39 EDT, Homer City-Watercure Road 
345-kV 

Homer City-Stolle Road 345-kV: 16:10:39 
EDT 

6E) South Ripley-Erie East 230-kV, and South 
Ripley-Dunkirk 230-kV: 16:10:44 EDT 
EastTowanda-Hillside 230-kV: 16:10:44 EDT 

Responding to the surge of power flowing north 
out of Pennsylvania through New York and 
Ontario into Michigan, relays on these lines acti- 
vated on apparent impedance within a five-second 
period and separated Pennsylvania from New 
York (Figure 5.10). 

At this point, the northern part of the Eastern 
Interconnection (including eastern Michigan and 
northern Ohio) remained connected to the rest of 
the Interconnection at only two locations: (1) in 

Figure 5.10. Western Pennsylvania Separates from 



Figure 5.11. More Transmission Line and Power 
Plant Losses 



the east through the 500-kV and 230-kV ties 
between New York and northeast New Jersey, and 
(2) in the west through the long and therefore frag- 
ile 230-kV transmission path connecting Ontario 
to Manitoba and Minnesota, 

Because the demand for power in Michigan, Ohio, 
and Ontario was drawing on lines through New 
York and Pennsylvania, heavy power flows were 
moving northward from New Jersey over the New 
York tie lines to meet those power demands, exac- 
erbating the power swing. 

6F) Conditions in Northern Ohio and Eastern 
Michigan Degraded Further, With More 
Transmission Lines and Power Plants Failing: 
16:10:39 to 16:10:46 EDT 

Bayshore-Monroe 345-kV line 

Allen Junction-Majestic-Monroe 345-kV line 

Majestic 345-kV Substation: one terminal 

opened on all 345-kV lines 

Perry- Ashtabula-Erie West 345-kV line terminal 

at Ashtabula 345/138-kV substation 

Fostoria Central-Galion 345-kV line 

Beaver-Da vis Besse 345-kV line 

Galion-Ohio Central-Muskingum 345 tripped at 

Galion 

Six power plants, for a total of 3,097 MW of gener- 
ation, tripped off-line: 

Lakeshore unit 18 (156 MW, near Cleveland) 
Bay Shore Units 1-4 (551 MW near Toledo) 
Eastlake 1, 2, and 3 units (403 MW total, near 
Cleveland) 

Avon Lake unit 9 (580 MW, near Cleveland) 
Perry 1 nuclear unit (1,223 MW, near 
Cleveland) 

Ashtabula unit 5 (184 MW, near Cleveland) 

Back in northern Ohio, the trips of the Majestic 
345-kV substation in southeast Michigan, the Bay 
Shore-Monroe 345-kV line, and the Ashtabula 
345/138-kV transformer created a Toledo and 
Cleveland electrical “island" (Figure 5.11). Fre- 
quency in this large island began to fall rapidly. 
This led to a series of power plants in the area 
shutting down due to the operation of under- 
frequency relays, including the Bay Shore units. 
When the Beaver-Davis Besse 345-kV line con- 
necting Cleveland and Toledo tripped, it left the 
Cleveland area completely isolated. Cleveland 
area load was disconnected by automatic under- 
frequency load-shedding (approximately 1,300 
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MW in the greater Cleveland area), and another 
434 MW of load was interrupted after the genera- 
tion remaining within this transmission “island’* 
was tripped by under-frequency relays. Portions 
of Toledo blacked out from automatic 
under-frequency load-shedding but most of the 
Toledo load was restored by automatic reclosing 
of lines such as the East Lima-Fostoria Central 
345-kV line and several lines at the Majestic 
345-kV substation. 

The prolonged period of system-wide low voltage 
around Detroit caused the remaining generators in 
that area, then running at maximum mechanical 
output, to begin to pull out of synchronous opera- 
tion with the rest of the grid. Those plants raced 
ahead of system frequency with higher than nor- 
mal revolutions per second by each generator. But 
when voltage returned to near-normal, the genera- 
tor could not fully pull back its rate of revolutions, 
and ended up producing excessive temporary out- 
put levels, still out of step with the system. This is 
evident in Figure 5.9 (above), which shows at least 
two sets of generator “pole slips” by plants in the 
Detroit area between 16:10:40 EDT and 16:10:42 
EDT. Several large units around Detroit — Belle 
River, St. Clair, Greenwood, Monroe and Fermi — 
all recorded tripping for out-of-step operation due 
to this cause. The Perry 1 nuclear unit, located on 
the southern shore of Lake Erie near the border 
with Pennsylvania, and a number of other units 
near Cleveland tripped off-line by unit under- 
frequency protection. 

6G) Transmission paths disconnected in New 
Jersey and northern Ontario, isolating the 
northeast portion of the Eastern 
Interconnection: 16:10:42 EDT to 16:10:45 EDT 

Four power plants producing 1,630 MW tripped 
off-line 

Greenwood unit 11 and 12 tripped (225 MW 
near Detroit) 

Belle River unit 1 tripped (600 MW near 
Detroit) 

St. Clair unit 7 tripped (221 MW, DTE unit) 
Trenton Channel units 7 A, 8 and 9 tripped 
(584 MW, DTE units) 

Keith-Waterman 230-kV tripped, 16:10:43 EDT 
Wawa-Marathon W21-22 230-kV line tripped, 
16:10:45 EDT 

Branchburg-Ramapo 500-kV line tripped, 
16:10:45 EDT 


A significant amount of the remaining generation 
serving Detroit tripped off-line in response to 
these events. At 16:10:43 EDT, eastern Michigan 
was still connected to Ontario, but the Keith- 
Waterman 230-kV line that forms part of that 
interface disconnected due to apparent imped- 
ance (Figure 5.12). 

At 16:10:45 EDT, northwest Ontario separated 
from the rest of Ontario when the Wawa-Marathon 
230-kV lines disconnected along the northern 
shore of Lake Superior. This separation left the 
loads in the far northwest portion of Ontario con- 
nected to the Manitoba and Minnesota systems, 
and protected them from the blackout. 

The Branchburg-Ramapo 500-kV line between 
New Jersey and New York was the last major trans- 
mission path remaining between the Eastern Inter- 
connection and the area ultimately affected by the 
blackout. That line disconnected at 16:10:45 EDT 
along with the underlying 230 and 138-kV lines 
in northeast New Jersey. This left the northeast 
portion of New Jersey connected to New York, 
while Pennsylvania and the rest of New Jersey 
remained connected to the rest of the Eastern 
Interconnection. 

At this point, the Eastern Interconnection was 
split into two major sections. To the north and east 
of the separation point lay New York City, north- 
ern New Jersey, New York slate, New England, the 
Canadian Maritime provinces, eastern Michigan, 
the majority of Ontario, and the Quebec system. 
The rest of the Eastern Interconnection, to the 
south and west of the separation boundary, was 
not seriously affected by the blackout. 


Figure 5.12. Northeast Disconnects from Eastern 
Interconnection 
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Phase 7: 

Several Electrical Islands Formed 
in Northeast U.S. and Canada: 
16:10:46 EDT to 16:12 EDT 

Overview of This Phase 

New England {except southwestern Connecticut) 
and the Marilimes separated from New York and 
remained intact; New York split east to west: 
16:10:46 EDT to 16:11:57 EDT. Figure 5.13 illus- 
trates the events of this phase. 

During the next 3 seconds, the islanded northern 
section of the Eastern Interconnection broke apart 
internally. 

7A) New York -New England transmission lines 
disconnected: 16:10:46 EDT to 16:10:47 EDT 

7B) 16:10:49 EDT, New York transmission sys- 
tem split east to west 

7C) The Ontario system just west of Niagara Falls 
and west of St. Lawrence separated from the 
western New York island: 16:10:50 EDT 

7D) Southwest Connecticut separated from New 
York City: 16:11:22 EDT 

7E) Remaining transmission lines between 
Ontario and eastern Michigan separated: 
16:11:57 EDT 

Key Phase 7 Events 

7 A) New York-New England Transmission 
lines Disconnected: 16:10:46 EDT to 16:10:49 
EDT 

Over the period 16:10:46 EDT to 16:10:49 EDT, the 
New York to New England tie lines tripped. The 
power swings continuing through the region 
caused this separation, and caused Vermont to 
lose approximately 70 MW of load. 

The lies between New York and New England dis- 
connected, and most of the New England area 
along with Canada’s Maritime Provinces became 
an island with generation and demand balanced 
close enough that it was able to remain opera- 
tional. New England had been exporting close to 
600 MW to New York, and its system experienced 
continuing fluctuations until it reached electrical 
equilibrium. Before the Maritimes-New England 
separated from the Eastern Interconnection at 
approximately 16:11 EDT, voltages became 
depressed due to the large power swings across 


portions of New England. Some large customers 
disconnected themselves automatically. 2 How- 
ever, southwestern Connecticut separated from 
New England and remained lied to the New York 
system for about 1 minute. 

Due to its geography and electrical characteristics, 
the Quebec system in Canada is tied to the remain- 
der of the Eastern Interconnection via high voltage 
DC links instead of AC transmission lines. Quebec 
was able to survive the power surges with only 
small impacts because the DC connections 
shielded it from the frequency swings. 

7B) New York Transmission Split East-West: 
16:10:49 EDT 

The transmission system split internally within 
New York, with the eastern portion islanding to 
contain New York City, northern New Jersey and 
southwestern Connecticut. The western portion of 
New York remained connected to Ontario and 
eastern Michigan. 

7C) The Ontario System fust West of Niagara 
Fcdls and West of St. Lawrence Separated from 
the Western New York Island: 16:10:50 EDT 

At 16:10:50 EDT, Ontario and New York separated 
west of the Ontario/New York interconnection, 
due to relay operations which disconnected nine 
230-kV lines within Ontario. These left most of 
Ontario isolated to the north. Ontario's large Beck 
and Saunders hydro stations, along with some 
Ontario load, the New York Power Authority’s 
(NYPA) Niagara and St. Lawrence hydro stations, 
and NYPA’s 765 -kV AC interconnection with 
Quebec, remained connected to the western New 
York system, supporting the demand in upstate 
New York. 


Figure 5.13. New York and New England Separate, 
Multiple Islands Form 
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From 16:10:49 EDT to 16:10:50 EDT, frequency 
declined below 59.3 Hz, initiating automatic 
under-frequency load-shedding in Ontario (2,500 
MW), eastern New York and southwestern Con- 
necticut. This load-shedding dropped off about 
20% of the load across the eastern New York 
island and about 10% of Ontario’s remaining load. 
Between 16:10:50 EDT and 16:10:56 EDT, the iso- 
lation of the southern Ontario hydro units onto the 
western New York island, coupled with 
under-frequency load-shedding in the western 
New York island, caused the frequency in this 
island to rise to 63.0 Hz due to excess generation. 

Three of the tripped 230-kV transmission circuits 
near Niagara automatically reconnected Ontario 
to New York at 16:10:56 EDT by reclosing. Even 
with these lines reconnected, the main Ontario 
island (still attached to New York and eastern 
Michigan) was then extremely deficient in genera- 
tion, so its frequency declined towards 58.8 Hz, 
the threshold for the second stage of under- 
frequency load-shedding. Within the next two sec- 
onds another 18% of Ontario demand (4,500 MW) 
automatically disconnected by under-frequency 
load-shedding. At 16:11:10 EDT, these same three 
lines tripped a second time west of Niagara, and 
New York and most of Ontario separated for a final 
time. Following this separation, the frequency in 
Ontario declined to 56 Hz by 16:11:57 EDT. With 
Ontario still supplying 2,500 MW to the Michi- 
gan-Ohio load pocket, the remaining ties with 
Michigan tripped at 16:11:57 EDT. Ontario system 
frequency declined, leading to a widespread shut- 
down at 16:11:58 EDT and loss of 22,500 MW of 


load in Ontario, including the cities of Toronto, 
Hamilton and Ottawa. 

7D ) Southwest Connecticut Separated from 
New York City: 16:11:22 EDT 

In southwest Connecticut, when the Long Moun- 
tain-Plum Tree line (connected to the Pleasant 
Valley substation in New York) disconnected at 
16 : 11:22 EDT, it left about 500 MW of southwest 
Connecticut demand supplied only through a 
138-kV underwater tie to Long Island. About two 
seconds later, the two 345-kV circuits connecting 
southeastern New York to Long Island tripped, 
isolating Long Island and southwest Connecticut, 
which remained tied together by the underwater 
Norwalk Harbor to Northport 138-kV cable. The 
cable tripped about 20 seconds later, causing 
southwest Connecticut to black out. 

Within the western New York island, the 345-kV 
system remained intact from Niagara east to the 
Utica area, and from the St. Lawrence/Plattsburgh 
area south to the Utica area through both the 
765-kV and 230-kV circuits. Ontario’s Beck and 
Saunders generation remained connected to New 
York at Niagara and St. Lawrence, respectively, 
and this island stabilized with about 50% of the 
pre-event load remaining. The boundary of this 
island moved southeastward as a result of the 
reclosure of Fraser to Coopers Corners 345-kV at 
16:11:23 EDT. 

As a result of the severe frequency and voltage 
changes, many large generating units in New York 
and Ontario tripped off-line. The eastern island of 
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New York, including the heavily populated areas 
of southeastern New York, New York City, and 
Long Island, experienced severe frequency and 
voltage decline. At 16:11:29 EDT, the New Scot- 
land to Leeds 345-kV circuits tripped, separating 
the island into northern and southern sections. 
The small remaining load in the northern portion 
of the eastern island (the Albany area) retained 
electric service, supplied by local generation until 
it could be resynchronized with the western New 
York island. 

7E) Remaining Trcmsmission Lines Between 
Ontario and Eastern Michigan Separated: 
16:11:57 EDT 

Before the blackout. New England, New York, 
Ontario, eastern Michigan, and northern Ohio 
were scheduled net importers of power. When the 
western and southern lines serving Cleveland, 
Toledo, and Detroit collapsed, most of the load 
remained on those systems, but some generation 
had tripped. This exacerbated the generation/load 
imbalance in areas that were already importing 
power. The power to serve this load came through 
the only major path available, through Ontario 
(IMO). After most of IMO was separated from New 
York and generation to the north and east, much of 
the Ontario load and generation was lost; it took 
only moments for the transmission paths west 
from Ontario to Michigan to fail. 

When the cascade was over at about 16:12 EDT, 
much of the disturbed area was completely 
blacked out, but there were isolated pockets that 
still had service because load and generation had 
reached equilibrium. Ontario's large Beck and 
Saunders hydro stations, along with some Ontario 
load, the New York Power Authority’s (NYPA) 
Figure 5.14. Electric Islands Reflected in 


Frequency Plot 



Niagara and St. Lawrence hydro stations, and 
NYPA's 765-kV AC interconnection with Quebec, 
remained connected to the western New York sys- 
tem, supporting demand in upstate New York. 

Electrical islanding. Once the northeast became 
isolated, it grew generation-deficient as more and 
more power plants tripped off-line to protect 
themselves from the growing disturbance. The 
severe swings in frequency and voltage in the area 
caused numerous lines to trip, so the isolated area 
broke further into smaller islands. The load/gener- 
ation mismatch also affected voltages and fre- 
quency within these smaller areas, causing further 
generator trips and automatic under-frequency 
load-shedding, leading to blackout in most of 
these areas. 

Figure 5.14 shows frequency data collected by the 
distribution-level monitors of Softswitching Tech- 
nologies, Inc. (a commercial power quality com- 
pany serving industrial customers) for the area 
affected by the blackout. The data reveal at least 
five separate electrical islands in the Northeast as 
the cascade progressed. The two paths of red dia- 
monds on the frequency scale reflect the Albany 
area island (upper path) versus the New York city 
island, which declined and blacked out much 
earlier. 

Cascading Sequence Essentially Complete: 
16:13 EDT 

Most of the Northeast (the area shown in gray in 
Figure 5.15) was now blacked out. Some isolated 
areas of generation and load remained on-line for 
several minutes. Some of those areas in which a 
close generation-demand balance could be main- 
tained remained operational; other generators ulti- 
mately tripped off line and the areas they served 
were blacked out. 


Figure 5.15. Area Affected by the Blackout 
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One relatively large island remained in operation 
serving about 5,700 MW of demand, mostly in 
western New York. Ontario’s large Beck and 
Saunders hydro stations, along with some Ontario 
load, the New York Power Authority’s (NYPA) 
Niagara and St. Lawrence hydro stations, and 
NYPA’s 765-kV AG interconnection with Quebec, 
remained connected to the western New York sys- 
tem, supporting demand in upstate New York. 
This island formed the basis for restoration in both 
New York and Ontario. 

The entire cascade sequence is depicted graphi- 
cally in Figure 5.16 on the following page. 

Why the Blackout Stopped 
Where It Did 

Extreme system conditions can damage equip- 
ment in several ways, from melting aluminum 
conductors (excessive currents) to breaking tur- 
bine blades on a generator (frequency excursions). 
The power system is designed to ensure that 
if conditions on the grid (excessive or inadequate 
voltage, apparent impedance or frequency) 
threaten the safe operation of the transmission 
lines, transformers, or power plants, the threat- 
ened equipment automatically separates from the 
network to protect itself from physical damage. 
Relays are the devices that effect this protection. 

Generators are usually the most expensive units 
on an electrical system, so system protection 
schemes are designed to drop a power plant off the 
system as a self-protective measure if grid condi- 
tions become unacceptable. When unstable power 
swings develop between a group of generators that 
are losing synchronization (matching frequency) 
with the rest of the system, the only way to stop 
the oscillations is to stop the flows entirely by sep- 
arating all interconnections or ties between the 
unstable generators and the remainder of the sys- 
tem. The most common way to protect generators 
from power oscillations is for the transmission 
system to detect the power swings and trip at the 
locations detecting the swings — ideally before the 
swing reaches and harms the generator. 

On August 14, the cascade became a race between 
the power surges and the relays. The lines that 
tripped first were generally the longer lines, 
because the relay settings required to protect these 
lines use a longer apparent impedance tripping 
zone, which a power swing enters sooner, in com- 
parison to the shorter apparent impedance zone 


targets set on shorter, networked lines. On August 
14, relays on long lines such as the Homer 
City-Watercure and the Homer City-Siolle Road 
345-kV lines in Pennsylvania, that are not highly 
integrated into the electrical network, tripped 
quickly and split the grid between the sections 
that blacked out and those that recovered without 
further propagating the cascade. This same phe- 
nomenon was seen in the Pacific Northwest black- 
outs of 1996, when long lines tripped before more 
networked, electrically supported lines. 

Transmission line voltage divided by its current 
flow is called “apparent impedance.” Standard 
transmission line protective relays continuously 
measure apparent impedance. When apparent 
impedance drops within the line’s protective relay 
set-points for a given period of time, the relays trip 
the line. The vast majority of trip operations on 
lines along the blackout boundaries between PJM 
and New York (for instance) show high-speed 
relay targets, which indicate that massive power 
surges caused each line to trip. To the relays, this 
massive power surge altered the voltages and cur- 
rents enough that they appeared to be faults. This 
power surge was caused by power flowing to those 
areas that were generation-deficient. These flows 
occurred purely because of the physics of power 
flows, with no regard to whether the power flow 
had been scheduled, because power flows from 
areas with excess generation into areas that are 
generation-deficient. 

Relative voltage levels across the northeast 
affected which areas blacked out and which areas 
stayed on-line. Within the Midwest, there were 
relatively low reserves of reactive power, so as 
voltage levels declined many generators in the 
affected area were operating at maximum reactive 
power output before the blackout. This left the 
system little slack to deal with the low voltage con- 
ditions by ramping up more generators to higher 
reactive power output levels, so there was little 
room to absorb any system “bumps” in voltage or 
frequency. In contrast, in the northeast— particu- 
larly PJM, New York, and ISO-New England — 
operators were anticipating high power demands 
on the afternoon of August 14, and had already set 
up the system to maintain higher voltage levels 
and therefore had more reactive reserves on-line 
in anticipation of later afternoon needs. Thus, 
when the voltage and frequency swings began, 
these systems had reactive power already or 
readily available to help buffer their areas against 
a voltage collapse without widespread generation 
trips. 
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Voltage Collapse 

Although the blackout of August 14 has been 
labeled as a voltage collapse, it was not a voltage 
collapse as that term has been traditionally used 
by power system engineers. Voltage collapse typi- 
cally occurs on power systems that are heavily 
loaded, faulted (reducing the number of available 
paths for power to flow to loads), or have reactive 
power shortages. The collapse is initiated when 
reactive power demands of loads can no longer be 
met by the production and transmission of reac- 
tive power. A classic voltage collapse occurs when 
an electricity system experiences a disturbance 
that causes a progressive and uncontrollable 
decline in voltage. Dropping voltage causes a fur- 
ther reduction in reactive power from capacitors 
and line charging, and still further voltage reduc- 
tions. If the collapse continues, these voltage 
reductions cause additional elements to trip, lead- 
ing to further reduction in voltage and loss of load. 
At some point the voltage may stabilize but at a 
much reduced level. In summary, the system 
begins to fail due to inadequate reactive power 
supplies rather than due to overloaded facilities. 

On August 14, the northern Ohio electricity sys- 
tem did not experience a classic voltage collapse 
because low voltage never became the primary 
cause of line and generator tripping. Although 
voltage was a factor in some of the events that led 
to the ultimate cascading of the system in Ohio 
and beyond, the event was not a classic reactive 
power-driven voltage collapse. Rather, although 
reactive power requirements were high, voltage 
levels were within acceptable bounds-before indi- 
vidual transmission trips began, and a shortage of 
reactive power did not trigger the collapse. Voltage 
levels began to degrade, but not collapse, as early 
transmission lines were lost due to tree-line con- 
tacts causing ground faults. With fewer lines oper- 
ational, current flowing over the remaining lines 
increased and voltage decreased (current in- 
creases in inverse proportion to the decrease in 
voltage for a given amount of power flow). Soon, 
in northern Ohio, lines began to trip out automati- 
cally on protection from overloads, rather than 
from insufficient reactive power. As the cascade 
spread beyond Ohio, it spread due not to insuffi- 
cient reactive power, but because of dynamic 
power swings and the resulting system instability. 

On August 14, voltage collapse in some areas was 
a result, rather than a cause, of the cascade. Signif- 
icant voltage decay began after the system was 
already in an N-3 or N-4 contingency situation. 


Frequency plots over the course of the cascade 
show areas with too much generation and others 
with too much load as the system attempted to 
reach equilibrium between generation and load. 
As the transmission line failures caused load to 
drop off, some parts of the system had too much 
generation, and some units tripped off on 
over-frequency protection. Frequency fell, more 
load dropped on under-frequency protection, the 
remaining generators sped up and then some of 
them tripped off, and so on. For a period, condi- 
tions see-sawed across the northeast, ending with 
isolated pockets in which generation and load had 
achieved balance, and wide areas that had blacked 
out before an equilibrium had been reached. 

Why the Generators Tripped Off 

At least 263 power plants with more than 531 indi- 
vidual generating units shut down in the August 
14 blackout. These U.S. and Canadian plants can 
be categorized as follows: 

By reliability coordination area: 

♦ Hydro Quebec, 5 plants 

♦ Ontario, 92 plants 

♦ ISO-New England, 31 plants 

♦ MISO, 30 plants 

♦ New York ISO, 67 plants 

♦ PJM, 38 plants 
By type: 

-♦ Conventional steam units, 67 plants (39 coal) 

♦ Combustion turbines, 66 plants (36 combined 

cycle) 

♦ Nuclear, 10 plants — 7 U.S. and 3 Canadian, 
totaling 19 units (the nuclear unit outages are 
discussed in Chapter 7) 

♦ Hydro, 101 

♦ Other, 19 

There were three categories of generator 
shutdowns: 

1. Excitation system failures during extremely low 
voltage conditions on portions of the power 
system 

2. Plant control system actions afteT major distur- 
bances to in-plant thermal/mechanical systems 

3. Consequential tripping due to total system dis- 
connection or collapse. 

Examples of the three types of separation are dis- 
cussed below. 
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Excitation failures. The Eastlake 5 trip at 1:31 
p.m. was an excitation system failure — as voltage 
fell at the generator bus, the generator tried to 
increase its production of voltage on the coil (exci- 
tation) quickly. This caused the generator’s excita- 
tion protection scheme to trip the plant off to 
protect its windings and coils from over-heating. 
Several of the other generators which tripped 
early in the cascade came off under similar 
circumstances as excitation systems were over- 
stressed to hold voltages up. 

After the cascade was initiated, huge power 
swings across the tom transmission system and 
excursions of system frequency put all the units in 
their path through a sequence of major distur- 
bances that shocked several units into tripping. 
Plant controls had actuated fast governor action 
on several of these to turn back the throttle, then 
turn it forward, only to turn it back again as some 
frequencies changed several times by as much as 3 
Hz (about 100 times normal). Figure 5.17 is a plot 
of the MW output and frequency for one large unit 
that nearly survived the disruption but tripped 
when in-plant hydraulic control pressure limits 
were eventually violated. After the plant control 
system called for shutdown, the turbine control 
valves closed and the generator electrical output 
ramped down to a preset value before the field 
excitation tripped and the generator breakers 
opened to disconnect the unit from the system. 

Plant control systems. The second reason for 
power plant trips was actions or failures of plant 
control systems. One common cause in this cate- 
gory was a loss of sufficient voltage to in-plant 
loadsr-Some plants run their internal cooling and 


Figure 5.17. Events at One Large Generator During 
the Cascade 



processes (house electrical load) off the generator 
or off small, in-house auxiliary generators, while 
others take their power off the main grid. When 
large power swings or voltage drops reached these 
plants in the latter category, they tripped off-line 
because the grid could not supply the plant's 
in-house power needs reliably. 

Consequential trips. Most of the unit separations 
fell in the third category of consequential trip- 
ping — they tripped off-line in response to some 
outside condition on the grid, not because of any 
problem internal to the plant. Some generators 
became completely removed from all loads; 
because the fundamental operating principle of 
the grid is that load and generation must balance, 
if there was no load to be served the power plant 
shut down in response to over-speed and/or 
over-voltage protection schemes. Others were 
overwhelmed because they were among a few 
power plants within an electrical island, and were 
suddenly called on to serve huge customer loads, 
so the imbalance caused them to trip on 
under-frequency and/or under-voltage protection. 
A few were tripped by special protection schemes 
that activated on excessive frequency or loss of 
pre-studied major transmission elements known 
to require large blocks of generation rejection. 

The maps in Figure 5.18 show the sequence of 
power plants lost in three blocks of time during 
the cascade. 

The investigation team is still analyzing data on 
the effect of the cascade on the affected generators, 
to learn more about how to protect generation and 
transmission assets and speed system restoration 
in the future. 

Endnotes 

1 The extensive computer modeling required to determine the 
expansion and cessation of the blackout (line by line, relay by 
relay, generator by generator, etc.) has not been performed. 
2 After New England’s separation from the Eastern Intercon- 
nection occurred, the next several minutes were critical to 
stabilizing the ISO-NE system. Voltages in New England 
recovered and over-shot to high due to the combination of 
load loss, capacitors still in service, lower reactive losses on 
the transmission system, and loss of generation to regulate 
system voltage. Over-voltage protective relays operated to trip 
both transmission and distribution capacitors. Operators in 
New England brought all fast-start generation on-line by 
16:16 EDT. Much of the customer process load was automati- 
cally restored. This caused voltages to drop again, putting 
portions of New England at risk of voltage collapse. Operators 
manually dropped 80 MW of load in southwest Connecticut 
by 16:39 EDT, another 325 MW in Connecticut and 100 MW 
in western Massachusetts by 16:40 EDT. These measures 
helped to stabilize their island following their separation 
from the rest of the Eastern Interconnection. 
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Figure 5.18. Power Plants Tripped During the Cascade 


O Generator Trtp6 Before 16:10:4 
• Generator Trips After 16:10:44 
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6. The August 14 Blackout Compared With 
Previous Major North American Outages 


Incidence and Characteristics 
of Power System Outages 

Short, localized outages occur on power systems 
fairly frequently. System-wide disturbances that 
affect many customers across a broad geographic 
area are rare, but they occur more frequently than 
a normal distribution of probabilities would pre- 
dict. North American power system outages 
between 1984 and 1997 are shown in Figure 6.1 by 
the number of customers affected and the rate of 
occurrence. While some of these were widespread 
weather-related events, some were cascading 
events that, in retrospect, were preventable. Elec- 
tric power systems are fairly robust and are capa- 
ble of withstanding one or two contingency 
events, but they are fragile with respect to multi- 
ple contingency events unless the systems are 
readjusted between contingencies. With the 
shrinking margin in the current transmission sys- 
tem, it is likely to be more vulnerable to cascading 
outages than it was in the past, unless effective 
countermeasures are taken. 

As evidenced by the absence of major transmis- 
sion projects undertaken iri'North America over 
the past 10 to 15 years, utilities have found ways to 
increase the utilization of their existing facilities 
to meet increasing demands without adding sig- 
nificant high-voltage equipment. Without inter- 
vention, this trend is likely to continue. Pushing 
the system harder will undoubtedly increase reli- 
ability challenges. Special protection schemes 
may be relied on more to deal with particular chal- 
lenges, but the system still will be less able to 
withstand unexpected contingencies. 

A smaller transmission margin for reliability 
makes the preservation of system reliability a 
harder job than it used to be. The system is being 
operated closer to the edge of reliability than it 
was just a few years ago. Table 6.1 represents some 
of the changed conditions that make the preserva- 
tion of reliability more challenging. 


Figure 6.1. North American Power System Outages, 
1984-1997 



Note: The bubbles represent individual outages in North 
America between 1984 and 1997. 


Source: Adapted from John Doyle, California Institute of 
Technology, “Complexity and Robustness,” 1999. Data from 
NERC. 

If nothing else changed, one could expect an 
increased frequency of large-scale events as com- 
pared to historical experience. The last and most 
extreme event shown in Figure 6.1 is the August 
10, 1996, outage. August 14, 2003, surpassed that 
event in terms of severity. In addition, two signifi- 
cant outages in the month of September 2003 
occurred abroad: one in England and one, initiated 
in Switzerland, that cascaded over much of Italy. 

In the following sections, seven previous outages 
are reviewed and compared with the blackout of 
August 14, 2003: (1) Northeast blackout on 
November 9, 1965; (2) New York City blackout on 
July 13, 1977; (3) West Coast blackout on Decem- 
ber 22, 1982; (4) West Coast blackout on July 2-3, 
1996; (5) West Coast blackout on August 10, 1996; 
(6) Ontario and U.S. North Central blackout on 
June 25, 1998; and {7J Northeast outages and non- 
outage disturbances in the summer of 1999. 
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Outage Descriptions 
and Major Causal Factors 

November 9, 1965: Northeast Blackout 

This disturbance resulted in the loss of over 
20,000 MW of load and affected 30 million people. 
Virtually all of New York, Connecticut, Massachu- 
setts, Rhode Island, small segments of northern 
Pennsylvania and northeastern New Jersey, and 
substantial areas of Ontario, Canada, were 
affected. Outages lasted for up to 13 hours. This 
event resulted in the formation of the North Amer- 
ican Electric Reliability Council in 1968. 

A backup protective relay operated to open one of 
five 230-kV lines taking power north from a gener- 
ating plant in Ontario to the Toronto area. When 
the flows redistributed instantaneously to the 
remaining four lines, they tripped out succes- 
sively in a total of 2.5 seconds. The resultant 
power swings resulted in a cascading outage that 
blacked out much of the Northeast. 

The major causal factors were as follows: 

♦ Operation of a backup protective relay took a 
230-kV line out of service when the loading on 
the line exceeded the 375-MW relay setting. 

♦ Operating personnel were not aware of the 
operating set point of this relay. 

♦ Another 230-kV line opened by an overcurrent 
relay action, and several 115- and 230-kV lines 
opened by protective relay action. 


♦ Two key 345-kV east-west (Rochester-Syracuse) 
lines opened due to instability, and several 
lower voltage lines tripped open. 

♦ Five of 16 generators at the St. Lawrence 
(Massena) plant tripped automatically in 
accordance with predetermined operating 
procedures. 

♦ Following additional line tripouts, 10 generat- 
ing units at Beck were automatically shut down 
by low governor oil pressure, and 5 pumping 
generators were tripped off by overspeed gover- 
nor control. 

♦ Several other lines then tripped out on 
under-frequency relay action. 

July 13, 1977: New York City Blackout 

This disturbance resulted in the loss of 6,000 MW 
of load and affected 9 million people in New York 
City. Outages lasted for up to 26 hours. A series of 
events triggering the separation of the Consoli- 
dated Edison system from neighboring systems 
and its subsequent collapse began when two 
345-kV lines on a common tower in Northern 
Westchester were struck by lightning and tripped 
out. Over the next hour, despite Consolidated Edi- 
son dispatcher actions, the system electrically 
separated from surrounding systems and col- 
lapsed. With the loss of imports, generation in 
New York City was not sufficient to serve the load 
in the city. 

Major causal factors were: 


Table 6.1. Changing Conditions That Affect System ReiiabHity 


Previous Conditions 

Emerging Conditions 

Fewer, relatively large resources 

Smaller, more numerous resources 

Long-term, firm contracts 

Contracts shorter in duration 

More non-firm transactions, fewer long-term firm transactions 

Bulk power transactions relatively stable and predictable 

Bulk power transactions relatively variable and less predictable 

Assessment of system reliability made from stable base 
(narrower, more predictable range of potential operating 
states) 

Assessment of system reliability made from variable base 
(wider, less predictable range of potential operating states) 

Limited and knowledgable set of utility players 

More players making more transactions, some with less 
interconnected operation experience; increasing with retail 
access 

Unused transmission capacity and high security margins 

High transmission utilization and operation closer to security 
limits 

Limited competition, little incentive for reducing reliability 
investments 

Utilities less willing to make investments in transmission 
reliability that do not increase revenues 

Market rules and reliability rules developed together 

Market rules undergoing transition, reliability rules developed 
separately 

Limited wheeling 

More system throughput 
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♦ Two 345-kV lines connecting Buchanan South 
to Millwood West were subjected to a phase B to 
ground fault caused by a lightning strike. 

♦ Circuit breaker operations at the Buchanan 
South ring bus isolated the Indian Point No. 3 
generating unit from any load, and the unit trip- 
ped for a rejection of 883 MW of load. 

♦ Loss of the ring bus isolated the 345-kV tie to 
Ladentown, which had been importing 427 
MW, making the cumulative load loss 1,310 
MW. 

♦ 18.5 minutes after the first incident, an addi- 
tional lightning strike caused the loss of two 
345-kV lines, which connect Sprain Brook to 
Buchanan North and Sprain Brook to Millwood 
West. These two 345-kV lines share common 
towers between Millwood West and Sprain 
Brook. One line (Sprain Brook to Millwood 
West) automatically reclosed and was restored 
to service in about 2 seconds. The failure of the 
other line to reclose isolated the last Consoli- 
dated Edison interconnection to the Northwest. 

♦ The resulting surge of power from the North- 
west caused the loss of the Pleasant Valley to 
Millwood West line by relay action (a bent con- 
tact on one of the relays at Millwood West 
caused the improper action). 

♦ 23 minutes later, the Leeds to Pleasant Valley 
345-kV line sagged into a tree due to overload 
and tripped out. 

♦ Within a minute, the 345 kV to 138 kV trans- 
former at Pleasant Valley overloaded and trip- 
ped off, leaving Consolidated Edison with only 
three remaining interconnections. 

♦ Within 3 minutes, the Long Island Lighting Co. 
system operator, on concurrence of the pool dis- 
patcher, manually opened the Jamaica to Valley 
Stream tie. 

♦ About 7 minutes later, the tap-changing mecha- 
nism failed on the Goethals phase-shifter, 
resulting in the loss of the Linden to Goethals tie 
to PJM, which was carrying 1,150 MW to Con- 
solidated Edison. 

♦ The two remaining external 138-kV ties to Con- 
solidated Edison tripped on overload, isolating 
the Consolidated Edison system. 

♦ Insufficient generation in the isolated system 
caused the Consolidated Edison island to 
collapse. 


December 22, 1982: West Coast Blackout 

This disturbance resulted in the loss of 12,350 
MW of load and affected over 5 million people in 
the West. The outage began when high winds 
caused the failure of a 500-kV transmission tower. 
The tower fell into a parallel 500-kV line tower, 
and both lines were lost. The failure of these two 
lines mechanically cascaded and caused three 
additional towers to fail on each line. When the 
line conductors fell they contacted two 230-kV 
lines crossing under the 500-kV rights-of-way, col- 
lapsing the 230-kV lines. 

The loss of the 500-kV lines activated a remedial 
action scheme to control the separation of the 
interconnection into two pre-engineered islands 
and trip generation in the Pacific Northwest in 
order to minimize customer outages and speed 
restoration. However, delayed operation of the 
remedial action scheme components occurred for 
several reasons, and the interconnection sepa- 
rated into four islands. 

In addition to the mechanical failure of the trans- 
mission lines, analysis of this outage cited prob- 
lems with coordination of protective schemes, 
because the generator tripping and separation 
schemes operated slowly or did not operate as 
planned. A communication channel component 
performed sporadically, resulting in delayed 
transmission of the control signal. The backup 
separation scheme also failed to operate, because 
the coordination of relay settings did not antici- 
pate the power flows experienced in this severe 
disturbance. 

In addition, the volume and format in which data 
were displayed to operators made it difficult to 
assess the extent of the disturbance and what cor- 
rective action should be taken. Time references to 
events in this disturbance were not tied to a com- 
mon standard, making real-time evaluation of the 
situation more difficult. 

July 2-3, 1996: West Coast Blackout 

This disturbance resulted in the loss of 11,850 
MW of load and affected 2 million people in the 
West. Customers were affected in Arizona, Cali- 
fornia, Colorado, Idaho, Montana, Nebraska, 
Nevada, New Mexico, Oregon, South Dakota, 
Texas, Utah, Washington, and Wyoming in the 
United States; Alberta and British Columbia in 
Canada; and Baja California Norte in Mexico. Out- 
ages lasted from a few minutes to several hours. 
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The outage began when a 345-kV transmission 
line in Idaho sagged into a tree and tripped out. A 
protective relay on a parallel transmission line 
also detected the fault and incorrectly tripped a 
second line. An almost simultaneous loss of these 
lines greatly reduced the ability of the system to 
transmit power from the nearby Jim Bridger plant. 
Other relays tripped two of the four generating 
units at that plant. With the loss of those two 
units, frequency in the entire Western Intercon- 
nection began to decline, and voltage began to col- 
lapse in the Boise, Idaho, area, affecting the 
California-Oregon AC Intertie transfer limit. 

For 23 seconds the system remained in precarious 
balance, until the Mill Creek to Antelope 230-kV 
line between Montana and Idaho tripped by zone 
3 relay, depressing voltage at Summer Lake Sub- 
station and causing the intertie to slip out of syn- 
chronism. Remedial action relays separated the 
system into five pre-engineered islands designed 
to minimize customer outages and restoration 
times. Similar conditions and initiating factors 
were present on July 3; however, as voltage began 
to collapse in the Boise area, the operator shed 
load manually and contained the disturbance. 

August 10, 1996: West Coast Blackout 

This disturbance resulted in the loss of over 
28,000 MW of load and affected 7.5 million people 
in the West. Customers were affected in Arizona, 
California, Colorado, Idaho, Montana, Nebraska, 
Nevada, New Mexico, Oregon, South Dakota, 
Texas, Utah, Washington, and Wyoming in the 
United States; Alberta and British Columbia in 
Canada; and Baja California Norte in Mexico. Out- - 
ages lasted from a few minutes to as long as 9 
hours. 

Triggered by several major transmission line out- 
ages, the loss of generation from McNary Dam, and 
resulting system oscillations, the Western Inter- 
connection separated into four electrical islands, 
with significant loss of load and generation. Prior 
to the disturbance, the transmission system from 
Canada south through the Northwest into Califor- 
nia was heavily loaded with north-to-south power 
transfers. These flows were due to high Southwest 
demand caused by hot weather, combined with 
excellent hydroelectric conditions in Canada and 
the Northwest. 

Very high temperatures in the Northwest caused 
two lightly loaded transmission lines to sag into 
untrimmed trees and trip out. A third heavily 
loaded line also sagged into a tree. Its outage led to 


the overload and loss of additional transmission 
lines. General voltage decline in the Northwest 
and the loss of McNary generation due to incor- 
rectly applied relays caused power oscillations on 
the California to Oregon AC intertie. The intertie's 
protective relays tripped these facilities out and 
caused the Western Interconnection to separate 
into four islands. Following the loss of the first two 
lightly loaded lines, operators were unaware that 
the system was in an insecure state over the next 
hour, because new operating studies had not been 
performed to identify needed system adjustment. 

June 25, 1998: Ontario and U.S. North 
Central Blackout 

This disturbance resulted in the loss of 950 MW of 
load and affected 152,000 people in Minnesota, 
Montana, North Dakota, South Dakota, and Wis- 
consin in the United States; and Ontario, Mani- 
toba, and Saskatchewan in Canada. Outages lasted 
up to 19 hours. 

A lightning storm in Minnesota initiated a series of 
events, causing a system disturbance that affected 
the entire Mid-Continent Area Power Pool (MAPPJ 
Region and the northwestern Ontario Hydro sys- 
tem of the Northeast Power Coordinating Council. 
A 345-kV line was struck by lightning and tripped 
out. Underlying lower voltage lines began to over- 
load and trip out, further weakening the system. 
Soon afterward, lightning struck a second 345-kV 
line, taking it out of service as well. Following the 
outage of the second 345-kV line, the remaining 
lower voltage transmission lines in the area, 
became significantly overloaded, and relays took 
them out of service. This cascading removal of 
lines from service continued until the entire 
northern MAPP Region was separated from the 
Eastern Interconnection, forming three islands 
and resulting in the eventual blackout of the 
northwestern Ontario Hydro system. 

Summer of 1999: Northeast U.S. Outages 
and Non-outage Disturbances 

Load in the PJM system on July 6, 1999, was 
51,600 MW {approximately 5,000 MW above fore- 
cast). PJM used all emergency procedures (includ- 
ing a 5% voltage reduction) except manually 
tripping load, and imported 5,000 MW from exter- 
nal systems to serve the record customer demand. 
Load on July 19, 1999, exceeded 50,500 MW. PJM 
loaded all available eastern PJM generation and 
again implemented PJM emergency operating pro- 
cedures from approximately 12 noon into the eve- 
ning on both days. 
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During record peak loads, steep voltage declines 
were experienced on the bulk transmission sys- 
tem. Emergency procedures were implemented to 
prevent voltage collapse. Low voltage occurred 
because reactive demand exceeded reactive sup- 
ply. High reactive demand was due to high elec- 
tricity demand and high losses resulting from high 
transfers across the system. Reactive supply was 
inadequate because generators were unavailable 
or unable to meet rated reactive capability due to 
ambient conditions, and because some shunt 
capacitors were out of service. 

Common or Similar Factors 
Among Major Outages 

Among the factors that were either common to the 
major outages above and the August 14 blackout 
or had similarities among the events are the fol- 
lowing: (1) conductor contact with trees; (2) 
underestimation of dynamic reactive output of 
system generators; (3) inability of system opera- 
tors or coordinators to visualize events on the 
entire system; (4) failure to ensure that system 
operation was within safe limits; (5) lack of coordi- 
nation on system protection; (6) ineffective com- 
munication; (7) lack of “safety nets;” and (8) 
inadequate training of operating personnel. The 
following sections describe the nature of these fac- 
tors and list recommendations from previous 
investigations that are relevant to each. 

Conductor Contact With Trees 

This factor was an initiating trigger in several of 
the outages and a contributing factor in the sever- 
ity of several more. Unlike lightning strikes, for 
which system operators have fair storm-tracking 
tools, system operators generally do not have 
direct knowledge that a line has contacted a tree 
and faulted. They will sometimes test the line by 
trying to restore it to service, if that is deemed to be 
a safe operation. Even if it does go back into ser- 
vice, the line may fault and trip out again as load 
heats it up. This is most likely to happen when 
vegetation has not been adequately managed, in 
combination with hot and windless conditions. 

In some of the disturbances, tree contact 
accounted for the loss of more than one circuit, 
contributing multiple contingencies to the weak- 
ening of the system. Lines usually sag into 
right-of-way obstructions when the need to retain 
transmission interconnection is significant. High 


inductive load composition, such as air condition- 
ing or irrigation pumping, accompanies hot 
weather and places higher burdens on transmis- 
sion lines. Losing circuits contributes to voltage 
decline. Inductive load is unforgiving when volt- 
age declines, drawing additional reactive supply 
from the system and further contributing to volt- 
age problems. 

Recommendations from previous investigations 
include: 

♦ Paying special attention to the condition of 
rights-of-way following favorable growing sea- 
sons. Very wet and warm spring and summer 
growing conditions preceded the 1996 outages 
in the West. 

♦ Careful review of any reduction in operations 
and maintenance expenses that may contribute 
to decreased frequency of line patrols or trim- 
ming. Maintenance in this area should be 
strongly directed toward preventive rather than 
remedial maintenance. 

Dynamic Reactive Output of Generators 

Reactive supply is an important ingredient in 
maintaining healthy power system voltages and 
facilitating power transfers. Inadequate reactive 
supply was a factor in most of the events. Shunt 
capacitors and generating resources are the most 
significant suppliers of reactive power. Operators 
perform contingency analysis based on how 
power system elements will perform under vari- 
ous power system conditions. They determine and 
set transfer limits based on these analyses. Shunt 
capacitors are easy to model because -they are 
static. Modeling the dynamic reactive output of 
generators under stressed system conditions has 
proven to be more challenging. If the model is 
incorrect, estimating transfer limits will also be 
incorrect. 

In most of the events, the assumed contribution of 
dynamic reactive output of system generators was 
greater than the generators actually produced, 
resulting in more significant voltage problems. 
Some generators were limited in the amount of 
reactive power they produced by over-excitation 
limits, or necessarily derated because of high 
ambient temperatures. Other generators were con- 
trolled to a fixed power factor and did not contrib- 
ute reactive supply in depressed voltage 
conditions. Under-voltage load shedding is 
employed as an automatic remedial action in some 
interconnections to prevent cascading. 
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Recommendations from previous investigations 
concerning voltage support and reactive power 
management include; 

♦ Communicate changes to generator reactive 
capability limits in a timely and accurate man- 
ner for both planning and operational modeling 
purposes. 

♦ Investigate the development of a generator 
MVAr/voltage monitoring process to determine 
when generators may not be following reported 
MVAr limits. 

♦ Establish a common standard for generator 
steady-state and post-contingency (15-minute) 
MVAr capability definition; determine method- 
ology, testing, and operational reporting 
requirements. 

♦ Determine the generator service level agree- 
ment that defines generator MVAr obligation to 
help ensure reliable operations. 

♦ Periodically review and field test the reactive 
limits of generators to ensure that reported 
MVAr limits are attainable. 

♦ Provide operators with on-line indications of 
available reactive capability from each generat- 
ing unit or groups of generators, other VAr 
sources, and the reactive margin at all critical 
buses. This information should assist in the 
operating practice of maximizing the use of 
shunt capacitors during heavy transfers and 
thereby increase the availability of system 
dynamic reactive reserve. 

♦ For voltage instability problems, consider fast 
automatic capacitor insertion (both series and 
shunt), direct shunt reactor and load tripping, 
and under-voltage load shedding. 

♦ Develop and periodically review a reactive mar- 
gin against which system performance should 
be evaluated and used to establish maximum 
transfer levels. 

System Visibility Procedures and 
Operator Tools 

Each control area operates as part of a single syn- 
chronous interconnection. However, the parties 
with various geographic or functional responsibil- 
ities for reliable operation of the grid do not have 
visibility of the entire system. Events in neighbor- 
ing systems may not be visible to an operator or 
reliability coordinator, or power system data 
may be available in a control center but not be 


presented to operators or coordinators as informa- 
tion they can use in making appropriate operating 
decisions. 

Recommendations from previous investigations 
concerning visibility and tools include: 

♦ Develop communications systems and displays 
that give operators immediate information on 
changes in the status of major components in 
their own and neighboring systems. 

♦ Supply communications systems with uninter- 
ruptible power, so that information on system 
conditions can be transmitted correctly to con- 
trol centers during system disturbances. 

♦ In the control center, use a dynamic line loading 
and outage display board to provide operating 
personnel with rapid and comprehensive infor- 
mation about the facilities available and the 
operating condition of each facility in service. 

♦ Give control centers the capability to display to 
system operators computer-generated alterna- 
tive actions specific to the immediate situation, 
together with expected results of each action. 

♦ Establish on-line security analysis capability to 
identify those next and multiple facility outages 
that would be critical to system reliability from 
thermal, stability, and post-contingency voltage 
points of view. 

♦ Establish time-synchronized disturbance moni- 
toring to help evaluate the performance of the 
interconnected system under stress, and design 
appropriate controls to protect it. 

System Operation Within Safe Limits 

Operators in several of the events were unaware of 
the vulnerability of the system to the next contin- 
gency. The reasons were varied: inaccurate model- 
ing for simulation, no visibility of the loss of key 
transmission elements, no operator monitoring of 
stability measures (reactive reserve monitor, 
power transfer angle), and no reassessment of sys- 
tem conditions following the loss of an element 
and readjustment of safe limits. 

Recommendations from previous investigations 
include: 

♦ Following a contingency, the system must be 
returned to a reliable state within the allowed 
readjustment period. Operating guides must be 
reviewed to ensure that procedures exist to 
restore system reliability in the allowable time 
periods. 
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♦ Reduce scheduled transfers to a safe and pru- 
dent level until studies have been conducted to 
determine the maximum simultaneous transfer 
capability limits. 

♦ Reevaluate processes for identifying unusual 
operating conditions and potential disturbance 
scenarios, and make sure they are studied 
before they are encountered in real-time operat- 
ing conditions. 

Coordination of System Protection 
(Transmission and Generation Elements) 

Protective relays are designed to detect abnormal 
conditions and act locally to isolate faulted power 
system equipment from the system — both to pro- 
tect the equipment from damage and to protect the 
system from faulty equipment. Relay systems are 
applied with redundancy in primary and backup 
modes. If one relay fails, another should detect the 
fault and trip appropriate circuit breakers. Some 
backup relays have significant "reach," such that 
non-faulted line overloads or stable swings maybe 
seen as faults and cause the tripping of a line when 
it is not advantageous to do so. Proper coordina- 
tion of the many relay devices in an intercon- 
nected system is a significant challenge, requiring 
continual review and revision. Some relays can 
prevent resynchronizing, making restoration more 
difficult. 

System-wide controls protect the interconnected 
operation rather than specific pieces of equip- 
ment. Examples include controlled islanding to 
mitigate the severity of an inevitable disturbance 
and under-voltage or under-frequency load shed- 
ding. Failure to operate (or misoperation of) one or 
more relays as an event developed was a common 
factor in several of the disturbances. 

Recommendations developed after previous out- 
ages include: 

♦ Perform system trip tests of relay schemes peri- 
odically. At installation the acceptance test 
should be performed on the complete relay 
scheme in addition to each individual compo- 
nent so that the adequacy of the scheme is 
verified. 

♦ Continually update relay protection to fit 
changing system development and to incorpo- 
rate improved relay control devices. 

♦ Install sensing devices on critical transmission 
lines to shed load or generation automatically if 
the short-term emergency rating is exceeded for 


a specified period of time. The time delay 
should be long enough to allow the system oper- 
ator to attempt to reduce line loadings promptly 
by other means. 

♦ Review phase-angle restrictions that can pre- 
vent reclosing of major interconnections during 
system emergencies. Consideration should be 
given to bypassing synchronism-check relays to 
permit direct closing of critical interconnec- 
tions when it is necessary to maintain stability 
of the grid during an emergency. 

♦ Review the need for controlled islanding. Oper- 
ating guides should address the potential for 
significant generation/load imbalance within 
the islands. 

Effectiveness of Communications 

Under normal conditions, parties with reliability 
responsibility need to communicate important 
and prioritized information to each other in a 
timely way, to help preserve the integrity of the 
grid. This is especially important in emergencies. 
During emergencies, operators should be relieved 
of duties unrelated to preserving the grid. A com- 
mon factor in several of the events described 
above was that information about outages occur- 
ring in one system was not provided to neighbor- 
ing systems. 

Need for Safety Nets 

A safety net is a protective scheme that activates 
automatically if a pre-specified, significant con- 
tingency occurs. When activated, such schemes 
involve certain costs and inconvenience, but they 
can prevent some disturbances from getting out of 
control. These plans involve actions such as shed- 
ding load, dropping generation, or islanding, and 
in all cases the intent is to have a controlled out- 
come that is less severe than the likely uncon- 
trolled outcome. If a safety net had not been taken 
out of service in the West in August 1996, it would 
have lessened the severity of the disturbance from 
28,000 MW of load lost to less than 7,200 MW. (It 
has since been returned to service.) Safety nets 
should not be relied upon to establish transfer lim- 
its, however. 

Previous recommendations concerning safety nets 
include: 

♦ Establish and maintain coordinated programs 
of automatic load shedding in areas not so 
equipped, in order to prevent total loss of power 
in an area that has been separated from the 
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main network and is deficient in generation. 
Load shedding should be regarded as an insur- 
ance program, however, and should not be used 
as a substitute for adequate system design. 

♦ Install load-shedding controls to allow fast sin- 
gle-action activation of large-block load shed- 
ding by an operator. 

Training of Operating Personnel 

Operating procedures were necessary but not suf- 
ficient to deal with severe power system distur- 
bances in several of the events. Enhanced 
procedures and training for operating personnel 
were recommended. Dispatcher training facility 
scenarios with disturbance simulation were sug- 
gested as well. Operators tended to reduce sched- 
ules for transactions but were reluctant to call for 
increased generation — or especially to shed 
load — in the face of a disturbance that threatened 
to bring the whole system down. 

Previous recommendations concerning training 
include: 

♦ Thorough programs and schedules for operator 
training and retraining should be vigorously 
administered. 

♦ A full-scale simulator should be made available 
to provide operating personnel with “hands-on” 
experience in dealing with possible emergency 
or other system conditions. 

♦ Procedures and training programs for system 
operators should include anticipation, recogni- 
tion, and definition of emergency situations. 

♦ Written procedures and training materials 
should include criteria that system operators 
can use to recognize signs of system stress and 
mitigating measures to be taken before condi- 
tions degrade into emergencies. 

♦ Line loading relief procedures should not be 
relied upon when the system is in an insecure 


state, as these procedures cannot be imple- 
mented effectively within the required time 
frames in many cases. Other readjustments 
must be used, and the system operator must 
take responsibility to restore the system 
immediately, 

♦ Operators' authority and responsibility to take 
immediate action if they sense the system is 
starting to degrade should be emphasized and 
protected. 

♦ The current processes for assessing the poten- 
tial for voltage instability and the need to 
enhance the existing operator training pro- 
grams, operational tools, and annual technical 
assessments should be reviewed to improve the 
ability to predict future voltage stability prob- 
lems prior to their occurrence, and to mitigate 
the potential for adverse effects on a regional 
scale. 

Comparisons With the 
August 14 Blackout 

The blackout on August 14, 2003, had several 
causes or contributory factors in common with the 
earlier outages, including: 

♦ Inadequate vegetation management 

♦ Failure to ensure operation within secure limits 

♦ Failure to identify emergency conditions and 
communicate that status to neighboring 
systems 

♦ Inadequate operator training 

♦ Inadequate regional-scale visibility over the 
power system. 

New causal features of the August 14 blackout 
include: inadequate interregional visibility over 
the power system; dysfunction of a control area’s 
SCADA/EMS system; and lack of adequate backup 
capability to that system. 
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7. Performance of Nuclear Power Plants 
Affected by the Blackout 


Summary 

On August 14, 2003, the northeastern United 
States and Canada experienced a widespread elec- 
trical power outage affecting an estimated 50 mil- 
lion people. Nine U.S. nuclear power plants 
experienced rapid shutdowns (reactor trips) as a 
consequence of the power outage. Seven nuclear 
power plants in Canada operating at high power 
levels at the time of the event also experienced 
rapid shutdowns. Four other Canadian nuclear 
plants automatically disconnected from the grid 
due to the electrical transient but were able to con- 
tinue operating at a reduced power level and were 
available to supply power to the grid as it was 
restored by the transmission system operators. Six 
nuclear plants in the United States and one in 
Canada experienced significant electrical distur- 
bances but were able to continue generating elec- 
tricity. Non-nuclear generating plants in both 
countries also tripped during the event. Numerous 
other nuclear plants observed disturbances on the 
slectrical grid but continued to generate electrical 
power without interruption. 

The Nuclear Working Group (NWG) is one of the 
three Working Groups created to support the 
U.S.-Canada Power System Outage Task Force. 
The NWG was charged with identifying all rele- 
vant actions by nuclear generating facilities in 
connection with the outage. Nils Diaz, Chairman 
of the U.S. Nuclear Regulatory Commission (NRC) 
and Linda Keen, President and CEO of the Cana- 
dian Nuclear Safety Commission (CNSC) are 
co-chairs of the Working Group, with other mem- 
bers appointed from various State and federal 
agencies. 

During Phase I of the investigation, the NWG 
focused on collecting and analyzing data from 
sach plant to determine what happened, and 
whether any activities at the plants caused or con- 
tributed to the power outage or involved a signifi- 
cant safety issue. To ensure accuracy, NWG 
members coordinated their efforts with the 


Electric System Working Group (ESWG) and the 
Security Working Group (SWG). NRC and CNSC 
staff developed a set of technical questions to 
obtain data from the owners or licensees of the 
nuclear power plants that would enable their staff 
to review the response of the nuclear plant sys- 
tems in detail. The plant data was compared 
against the plant design to determine if the plant 
responses were as expected; if they appeared to 
cause the power outage or contributed to the 
spread of the outage; and if applicable safety 
requirements were met. 

Having reviewed the operating data for each plant 
and the response of the nuclear power plants and 
their staff to the event, the NWG concludes the 
following: 

♦ All the nuclear plants that shut down or discon- 
nected from the grid responded automatically 
to grid conditions. 

♦ All the nuclear plants responded in a manner 
consistent with the plant designs. 

♦ Safety functions were effectively accomplished, 
and the nuclear plants that tripped were main-_ 
tamed in a safe shutdown condition until their 
restart. 

♦ The nuclear power plants did not trigger the 
power system outage or inappropriately con- 
tribute to its spread (i.e., to an extent beyond the 
normal tripping of the plants at expected condi- 
tions). Rather, they responded as anticipated in 
order to protect equipment and systems from 
the grid disturbances. 

♦ For nuclear plants in the United States: 

>■ Fermi 2, Oyster Creek, and Perry tripped due 
to main generator trips, which resulted from 
voltage and frequency fluctuations on the 
grid. Nine Mile 1 tripped due to a main tur- 
bine trip due to frequency fluctuations on the 
grid. 
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> FitzPatrick and Nine Mile 2 tripped due to 
reactor trips, which, resulted from turbine 
control system low pressure due to frequency 
fluctuations on the grid. Ginna tripped due to 
a reactor trip which resulted from a large loss 
of electrical load due to frequency fluctua- 
tions on the grid. Indian Point 2 and Indian 
Point 3 tripped due to a reactor trip on low 
flow, which resulted when low grid fre- 
quency tripped reactor coolant pumps. 

♦ For nuclear plants in Canada: 

> At Bruce B and Pickering B, frequency and/or 
voltage fluctuations on the grid resulted in 
the automatic disconnection of generators 
from the grid. For those units that were suc- 
cessful in maintaining the unit generators 
operational, reactor power was automatically 
reduced. 

>- At Darlington, load swing on the grid led to 
the automatic reduction in power of the four 
reactors. The generators were, in turn, auto- 
matically disconnected from the grid. 

>> Three reactors at Bruce B and one at Darling- 
ton were returned to 60% power. These reac- 
tors were available to deliver power to the 
grid on the instructions of the transmission 
system operator. 

*» Three units at Darlington were placed in a 
zero-power hot state, and four units at 
Pickering B and one unit at Bruce B were 
placed in a Guaranteed Shutdown State. 

"The licensees’ return to power operation follows a 
deliberate process controlled by plant procedures 
and regulations. Equipment and process prob- 
lems, whether existing prior to or caused by the 
event, would normally be addressed prior to 
restart. The NWG is satisfied that licensees took an 
appropriately conservative approach to their 
restart activities, placing a priority on safety. 

♦ For U.S. nuclear plants: Ginna, Indian Point 2, 
Nine Mile 2, and Oyster Creek resumed electri- 
cal generation on August 17. FitzPatrick and 
Nine Mile 1 resumed electrical generation on 
August 18. Fermi 2 resumed electrical genera- 
tion on August 20. Perry resumed electrical gen- 
eration on August 21. Indian Point 3 resumed 
electrical generation on August 22. Indian Point 
3 had equipment issues (failed splices in the 
control rod drive mechanism power system) 
that required repair prior to restart. Ginna 
submitted a special request for enforcement 


discretion from the NRC to permit mode 
changes and restart with an inoperable auxil- 
iary feedwater pump. The NRC granted the 
request for enforcement discretion. 

♦ For Canadian nuclear plants: The restart of the 
Canadian nuclear plants was carried out in 
accordance with approved Operating Policies 
and Principles. Three units at Bruce B and one 
at Darlington were resynchronized with the grid 
within 6 hours of the event. The remaining 
three units at Darlington were reconnected by 
August 17 and 18. Units 5, 6, and 8 at Pickering 
B and Unit 6 at Bruce B returned to service 
between August 22 and August 25. 

The NWG has found no evidence that the shut- 
down of the nuclear power plants triggered the 
outage or inappropriately contributed to its spread 
(i.e., to an extent beyond the normal tripping of 
the plants at expected conditions). All the nuclear 
plants that shut down or disconnected from the 
grid responded automatically to grid conditions. 
All the nuclear plants responded in a manner con- 
sistent with the plant designs. Safety functions 
were effectively accomplished, and the nuclear 
plants that tripped were maintained in a safe shut- 
down condition until their restart. 

Additional details are available in the following 
sections. Due to the major design differences 
between nuclear plants in Canada and the United 
States, the decision was made to have separate 
sections for each country. This also facilitates the 
request by the nuclear regulatory agencies in both 
countries to have sections of the report that stand 
alone, so that they can also be used asjregulatory 
documents. 

Findings of the U.S. Nuclear 
Working Group 

Summary 

The U.S. NWG has found no evidence that the 
shutdown of the nine U.S. nuclear power plants 
triggered the outage, or inappropriately contrib- 
uted to its spread (i.e., to an extent beyond the nor- 
mal tripping of the plants at expected conditions). 
All nine plants that experienced a reactor trip 
were responding to grid conditions. The severity 
of the grid transient caused generators, turbines, 
or reactor systems at the plants to reach a protec- 
tive feature limit and actuate a plant shutdown. 
All nine plants tripped in response to those 
conditions in a manner consistent with the plant 
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designs. The nine plants automatically shut down 
in a safe fashion to protect the plants from the grid 
transient. Safety functions were effectively 
accomplished with few problems, and the plants 
were maintained in a safe shutdown condition 
until their restart. 

The nuclear power plant outages that resulted 
from the August 14, 2003, power outage were trig- 
gered by automatic protection systems for the 
reactors or turbine-generators, not by any manual 
operator actions. The NWG has received no infor- 
mation that points to operators deliberately shut- 
ting down nuclear units to isolate themselves from 
instabilities on the grid. In short, only automatic 
separation of nuclear units occurred. 

Regarding the 95 other licensed commercial 
nuclear power plants in the United States: 4 were 
already shut down at the time of the power outage, 
one of which experienced a grid disturbance; 70 
operating plants observed some level of grid dis- 
turbance but accommodated the disturbances and 
remained on line, supplying power to the grid; and 
21 operating plants did not experience any grid 
disturbance. 

Introduction 

In response to the August 14 power outage, the 
United States and Canada established a joint 
Power System Outage Task Force. Although many 
non-nuclear power plants were involved in the 
power outage, concerns about the nuclear power 
plants are being specifically addressed by the 
NWG in supporting of the joint Task Force._The 
Task Force was tasked with answering two 
questions: 

1. What happened on August 14, 2003, to cause 
the transmission system to fail resulting in the 
power outage, and why? 

2. Why was the system not able to stop the spread 
of the outage? 

The NRC, which regulates U.S. commercial 
nuclear power plants, has regulatory requirements 
for offsite power systems. These requirements 
address the number of offsite power sources and 
the ability to withstand certain transients. Offsite 
power is the normal source of alternating current 
(AC) power to the safety systems in the plants 
when the plant main generator is not in operation. 
The requirements also are designed to protect 
safety systems from potentially damaging varia- 
tions (in voltage and frequency) in the supplied 


power. For loss of offsite power events, the NRC 
requires emergency generation (typically emer- 
gency diesel generators) to provide AC power to 
safety systems. In addition, the NRC provides 
oversight of the safety aspects of offsite power 
issues through its inspection program, by moni- 
toring operating experience, and by performing 
technical studies. 

Phase I: Fact Finding 

Phase I of the NWG effort focused on collecting 
and analyzing data from each plant to determine 
what happened, and whether any activities at the 
plants caused or contributed to the power outage 
or its spread or involved a significant safety issue. 
To ensure accuracy, a comprehensive coordina- 
tion effort is ongoing among the working group 
members and between the NWG, ESWG, and 
SWG. 

The staff developed a set of technical questions to 
obtain data from the owners or licensees of the 
nuclear power plants that would enable them to 
review the response of the nuclear plant systems 
in detail. Two additional requests for more spe- 
cific information were made for certain plants. 
The collection of information from U.S. nuclear 
power plants was gathered through the NRC 
regional offices, which had NRC resident inspec- 
tors at each plant obtain licensee information to 
answer the questions. General design information 
was gathered from plant-specific Updated Final 
Safety Analysis Reports and other documents. 

Plant data were compared against plant designs by 
the NRC staff to determine whether the plant 
responses were as expected; whether they 
appeared to cause the power outage or contributed 
to the spread of the outage; and whether applica- 
ble safety requirements were met. In some cases 
supplemental questions were developed, and 
answers were obtained from the licensees to clar- 
ify the observed response of the plant. The NWG 
interfaced with the ESWG to validate some data 
and to obtain grid information, which contributed 
to the analysis. The NWG has identified relevant 
actions by nuclear generating facilities in connec- 
tion with the power outage. 

Typical Design, Operational, and 
Protective Features of U.S. Nuclear 
Power Plants 

Nuclear power plants have a number of design, 
operational, and protective features to ensure that 
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the plants operate safely and reliably. This section 
describes these features so as to provide a better 
understanding of how nuclear power plants inter- 
act with the grid and, specifically, how nuclear 
power plants respond to changing grid conditions. 
While the features described in this section are 
typical, there are differences in the design and 
operation of individual plants which are not 
discussed. 

Design Features of Nuclear Power Plants 

Nuclear power plants use heat from nuclear reac- 
tions to generate steam and use a single steam- 
driven turbine-generator (also known as the main 
generator) to produce electricity supplied to the 
grid. 

Connection of the plant switchyard to the grid. 
The plant switchyard normally forms the interface 
between the plant main generator and the electri- 
cal grid. The plant switchyard has multiple trans- 
mission lines connected to the grid system to meet 
offsite power supply requirements for having reli- 
able offsite power for the nuclear station under all 
operating and shutdown conditions. Each trans- 
mission line connected to the switchyard has ded- 
icated circuit breakers, with fault sensors, to 
isolate faulted conditions in the switchyard or the 
connected transmission lines, such as phase-to- 
phase or phase-to-ground short circuits. The fault 
sensors are fed into a protection scheme for the 
plant switchyard that is engineered to localize 
any faulted conditions with minimum system 
disturbance. 

Connection of the main generator to the switch- 
yard. The plant main generator produces electri- 
cal power and transmits that power to the offsite 
transmission system. Most plants also supply 
power to the plant auxiliary buses for normal 
operation of the nuclear generating unit through 
the unit auxiliary transformer. During normal 
plant operation, the main generator typically gen- 
erates electrical power at about 22 kV. The voltage 
is increased to match the switchyard voltage by 
the main transformers, and the power flows to the 
high voltage switchyard through two power cir- 
cuit breakers. 

Power supplies for the plant auxiliary buses. The 
safety-related and nonsafety auxiliary buses are 
normally lined up to receive power from the main 
generator auxiliary transformer, although some 
plants leave some of their auxiliary buses powered 
from a startup transformer (that is, from the offsite 
power distribution system). When plant power 
generation is interrupted, the power supply 


automatically transfers to the offsite power source 
(the startup transformer). If that is not supplying 
acceptable voltage, the circuit breakers to the 
safety-related buses open, and the buses are 
reenergized by the respective fast-starting emer- 
gency diesel generators. The nonsafety auxiliary 
buses will remain deenergized until offsite power 
is restored. 

Operational Features of Nuclear Power Plants 

Response of nuclear power plants to changes in 
switchyard voltage. With the main generator volt- 
age regulator in the automatic mode, the generator 
will respond to an increase of switchyard voltage 
by reducing the generator field excitation current. 
This will result in a decrease of reactive power, 
normally measured as mega-volts-amperes-reac- 
tive (MVAR) from the generator to the switchyard 
and out to the surrounding grid, helping to control 
the grid voltage increase. With the main generator 
voltage regulator in the automatic mode, the gen- 
erator will respond to a decrease of switchyard 
voltage by increasing the generator field excitation 
current. This will result in an increase of reactive 
power (MVAR) from the generator to the 
switchyard and out to the surrounding grid, help- 
ing to control the grid voltage decrease. If the 
switchyard voltage goes low enough, the 
increased generator field current could result in 
generator field overheating. Over-excitation pro- 
tective circuitry is generally employed to prevent 
this from occurring. This protective circuitry may 
trip the generator to prevent equipment damage. 

Under-voltage protection is provided for the 
nuclear power plant safety buses, and may be pro- 
vided on nonsafety buses and at individual pieces 
of equipment. It is also used in some pressurized 
water reactor designs on reactor coolant pumps 
(RCPs) as an anticipatory loss of RCP flow signal. 

Protective Features of Nuclear Power Plants 

The main generator and main turbine have protec- 
tive features, similar to fossil generating stations, 
which protect against equipment damage. In gen- 
eral, the reactor protective features are designed to 
protect the reactor fuel from damage and to protect 
the reactor coolant system from over-pressure or 
over-temperature transients. Some trip features 
also produce a corresponding trip in other compo- 
nents; for example, a turbine trip typically results 
in a reactor trip above a low power setpoint. 

Generator protective features typically include 
over-current, ground detection, differential relays 
(which monitor for electrical fault conditions 
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within a zone of protection defined by the location 
of the sensors, typically the main generator and all 
transformers connected directly to the generator 
output), electrical faults on the transformers con- 
nected to the generator, loss of the generator field, 
and a turbine trip. Turbine protective features typ- 
ically include over-speed (usually set at 1980 rpm 
or 66 Hz), low bearing oil pressure, high bearing 
vibration, degraded condenser vacuum, thrust 
bearing failure* or generator trip. Reactor protec- 
tive features typically include trips for over- 
power, abnormal pressure in the reactor coolant 
system, low reactor coolant system flow, low level 
in the steam generators or the reactor vessel, or a 
trip of the turbine. 

Considerations on Returning a U.S. 
Nuclear Power Plant to Power 
Production After Switchyard Voltage 
Is Restored 

The following are examples of the types of activi- 
ties that must be completed before returning a 
nuclear power plant to power production follow- 
ing a loss of switchyard voltage. 

♦ Switchyard voltage must be normal and stable 
from an offsite supply. Nuclear power plants are 
not designed for black-start capability (the abil- 
ity to start up without external power). 

♦ Plant buses must be energized from the 
switchyard and the emergency diesel genera- 
tors restored to standby mode. 

♦ Normal plant equipment, such as reactor cool- 
__ant pumps and circulating water pumps, must 

be restarted. 

♦ A reactor trip review report must be completed 
and approved by plant management, and the 
cause of the trip must be addressed. 

♦ All plant technical specifications must be satis- 
fied. Technical specifications are issued to each 
nuclear power plant as part of their license by 
the NRC. They dictate equipment which must 
be operable and process parameters which must 
be met to allow operation of the reactor. Exam- 
ples of actions that were required following the 
events of August 14 include refilling the diesel 
fuel oil storage tanks, refilling the condensate 
storage tanks, establishing reactor coolant sys- 
tem forced flow, and cooling the suppression 
pool to normal operating limits. Surveillance 
tests must be completed as required by techni- 
cal specifications (for example, operability of 


the low-range neutron detectors must be 
demonstrated). 

♦ Systems must be aligned to support the startup. 

♦ Pressures and temperatures for reactor startup 
must be established in the reactor coolant sys- 
tem for pressurized water reactors. 

♦ A reactor criticality calculation must be per- 
formed to predict the control rod withdrawals 
needed to achieve criticality, where the fission 
chain reaction becomes self-sustaining due to 
the increased neutron flux. Certain neutron- 
absorbing fission products increase in concen- 
tration following a reactor trip (followed later 
by a decrease or decay). At pressurized water 
reactors, the boron concentration in the primary 
coolant must be adjusted to match the criticality 
calculation. Near the end of the fuel cycle, the 
nuclear power plant may not have enough 
boron adjustment or control rod worth available 
for restart until the neutron absorbers have 
decreased significantly (more than 24 hours 
after the trip). 

It may require about a day or more before a nuclear 
power plant can restart following a normal trip. 
Plant trips are a significant transient on plant 
equipment, and some maintenance may be neces- 
sary before the plant can restart. When combined 
with the infrequent event of loss of offsite power, 
additional recovery actions will be required. 
Safety systems, such as emergency diesel genera- 
tors and safety-related decay heat removal sys- 
tems, must be restored to normal lineups. These 
additional actions would extend the time neces- 
sary~to restart a nuclear plant from this type of 
event. 

Summary of U.S. Nuclear Power Plant 
Response to and Safety During the 
August 14 Outage 

The NWG’s review has not identified any activity 
or equipment issues at nuclear power plants that 
caused the transient on August 14, 2003. Nine 
nuclear power plants tripped within about 60 sec- 
onds as a result of the grid disturbance. Addi- 
tionally, many nuclear power plants experienced 
a transient due to this grid disturbance. 

Nuclear Power Plants That Tripped 

The trips at nine nuclear power plants resulted 
from the plant responses to the grid disturbances. 
Following the initial grid disturbances, voltages in 
the plant switchyard fluctuated and reactive 
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power flows fluctuated. As the voltage regulators 
on the main generators attempted to compensate, 
equipment limits were exceeded and protective 
trips resulted. This happened at Fermi 2 and Oys- 
ter Creek. Fermi 2 tripped on a generator field pro- 
tection trip. Oyster Creek tripped due to a 
generator trip on high ratio of voltage relative to 
the electrical frequency. 

Also, as the balance between electrical generation 
and electrical load on the grid was disturbed, the 
electrical frequency began to fluctuate. In some 
cases the electrical frequency dropped low 
enough to actuate protective features. This hap- 
pened at Indian Point 2, Indian Point 3, and Perry. 
Perry tripped due to a generator under-frequency 
trip signal. Indian Point 2 and Indian Point 3 trip- 
ped when the grid frequency dropped low enough 
to trip reactor coolant pumps, which actuated a 
reactor protective feature. 

In other cases, the electrical frequency fluctuated 
and went higher than normal. Turbine control sys- 
tems responded in an attempt to control the fre- 
quency. Equipment limits were exceeded as a 
result of the reaction of the turbine control sys- 
tems to large frequency changes. This led to trips 
at FitzPatrick, Nine Mile 1, Nine Mile 2, and 
Ginna. FitzPatrick and Nine Mile 2 tripped on low 
pressure in the turbine hydraulic control oil sys- 
tem. Nine Mile 1 tripped on turbine light load pro- 
tection. Ginna tripped due to conditions in the 
reactor following rapid closure of the turbine con- 
trol valves in response to high frequency on the 
grid. 

The Perry7 Fermi 2, Oyster Creek, and Nine Mile 1 
reactors tripped immediately after the generator 
tripped, although that is not apparent from the 
times below, because the clocks were not synchro- 
nized to the national time standard. The Indian 
Point 2 and 3, FitzPatrick, Ginna, and Nine Mile 2 
reactors tripped before the generators. When the 
reactor trips first, there is generally a short time 
delay before the generator output breakers open. 
The electrical generation decreases rapidly to zero 
after the reactor trip. Table 7.1 provides the times 
from the data collected for the reactor trip times, 
and the time the generator output breakers opened 
(generator trip), as reported by the ESWG. Addi- 
tional details on the plants that tripped are given 
below. 

Fermi 2. Fermi 2 is located 25 miles northeast of 
Toledo, Ohio, in southern Michigan on Lake Erie. 
It was generating about 1,130 megawaUs-electric 
(MWe) before the event. The reactor tripped due to 


a turbine trip. The turbine trip was likely the 
result of multiple generator field protection trips 
{over-excitation and loss of field} as the Fermi 2 
generator responded to a series of rapidly chang- 
ing transients prior to its loss. This is consistent 
with data that shows large swings of the Fermi 2 
generator MVARs prior to its trip. 

Offsite power was subsequently lost to the plant 
auxiliary buses. The safety buses were de- 
energized and automatically reenergized from the 
emergency diesel generators. The operators trip- 
ped one emergency diesel generator that was 
paralled to the grid for testing, after which it auto- 
matically loaded. Decay heat removal systems 
maintained the cooling function for the reactor 
fuel. 

The lowest emergency declaration, an Unusual 
Event, was declared at about 16:22 EDT due to the 
loss of offsite power. Offsite power was restored to 
at least one safety bus at about 01:53 EDT on 
August 15. The following equipment problems 
were noted: the Combustion Turbine Generator 
(the alternate AC power source) failed to start from 
the control room; however, it was successfully 
started locally. In addition, the Spent Fuel Pool 
Cooling System was interrupted for approxi- 
mately 26 hours and reached a maximum temper- 
ature of 130 degrees Fahrenheit (55 degrees 
Celsius). The main generator was reconnected to 
the grid at about 01:41 EDT on August 20. 

FitzPatrick. FitzPatrick is located about 8 miles 
northeast of Oswego, NY, in northern New York 
on Lake Ontario. It was generating about 850 MWe 
before the event. The reactor tripped due to low- 
pressure in the hydraulic system that controls the 
turbine control valves. Low pressure in this sys- 
tem typically indicates a large load reject, for 

Table 7.1. U.S. Nuclear Plant Trip Times 

Nuclear Plant | Reactor Trip* j Generator Trlp b 


Perry 16:10:25 EDT 16:10:42 EDT 

Fermi 2 16:10:53 EDT 16:10:53 EDT 

Oyster Creek. . . 16:10:58 EDT 16:10:57 EDT 

Nine Mile 1 .... 16:11 EDT 16:11:04 EDT 

Indian Point2 .. 16:11 EDT 16:11:09EDT 

Indian Point 3 . . 16:11 EDT 16:11:23 EDT 

FitzPatrick 16:11:04 EDT 16:11:32 EDT 

Ginna 16:11:36 EDT 16:12:17 EDT 

Nine Mile 2 . . . . 16:11:48EDT 16:11:52 EDT 


a As determined from licensee data (which may not be syn- 
chronized to the national time standard). 

b As reported by the Electrical System Working Group (syn- 
chronized to the national time standard). 
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which a reactor trip is expected. In this case the 
pressure in the system was low because the con- 
trol system was rapidly manipulating the turbine 
control valves to control turbine speed, which was 
being affected by grid frequency fluctuations. 

Immediately preceding the trip, both significant 
over-voltage and under-voltage grid conditions 
were experienced. Offsite power was subse- 
quently lost to the plant auxiliary buses. The 
safety buses were deenergized and automatically 
reenergized from the emergency diesel generators. 

The lowest emergency declaration, an Unusual 
Event, was declared at about 16:26 EDT due to the 
loss of offsite power. Decay heat removal systems 
maintained the cooling function for the reactor 
fuel. Offsite power was restored to at least one 
safety bus at about 23:07 EDT on August 14. The 
main generator was reconnected to the grid at 
about 06:10 EDT on August 18. 

Ginna. Ginna is located 20 miles northeast of 
Rochester, NY, in northern New York on Lake 
Ontario. It was generating about 487 MWe before 
the event. The reactor tripped due to Over- 
Temperature-Delta-Temperature. This trip signal 
protects the reactor core from exceeding tempera- 
ture limits. The turbine control valves closed 
down in response to the changing grid conditions. 
This caused a temperature and pressure transient 
in the reactor, resulting in an Over-Temperature- 
Delta-Temperature trip. 

Offsite power was not lost to the plant auxiliary 
buses. In the operators' judgement, offsite power 
was not stable, so they conservatively energized 
the safety buses from the emergency diesel genera- 
tors. Decay heat removal systems maintained the 
cooling function for the reactor fuel. Offsite power 
was not lost, and stabilized about 50 minutes after 
the reactor trip. 

The lowest emergency declaration, an Unusual 
Event, was declared at about 16:46 EDT due to the 
degraded offsite power. Offsite power was 
restored to at least one safety bus at about 21:08 
EDT on August 14. The following equipment 
problems were noted: the digital feedwater control 
system behaved in an unexpected manner follow- 
ing the trip, resulting in high steam generator lev- 
els; there was a loss of RCP seal flow indication, 
which complicated restarting the pumps; and at 
least one of the power-operated relief valves expe- 
rienced minor leakage following proper operation 
and closure during the transient. Also, one of the 
motor-driven auxiliary feedwater pumps was 


damaged after running with low flow conditions 
due to an improper valve alignment. The redun- 
dant pumps supplied the required water flow. 

The NRC issued a Notice of Enforcement Discre- 
tion to allow Ginna to perform mode changes and 
restart the reactor with one auxiliary feedwater 
(AFW) pump inoperable. Ginna has two AFW 
pumps, one turbine-driven AFW pump, and two 
standby AFW pumps, all powered from safety- 
related buses. The main generator was recon- 
nected to the grid at about 20:38 EDT on August 
17. 

Indian Point 2. Indian Point 2 is located 24 miles 
north of New York City on the Hudson River. It 
was generating about 990 MWe before the event. 
The reactor tripped due to loss of a reactor coolant 
pump that tripped because the auxiliary bus fre- 
quency fluctuations actuated the under-frequency 
relay, which protects against inadequate coolant 
flow through the reactor core. This reactor protec- 
tion signal tripped the reactor, which resulted in 
turbine and generator trips. 

The auxiliary bus experienced the under- 
frequency due to fluctuating grid conditions. 
Offsite power was lost to all the plant auxiliary 
buses. The safety buses were reenergized from the 
emergency diesel generators. Decay heat removal 
systems maintained the cooling function for the 
reactor fuel. 

The lowest emergency declaration, an Unusual 
Event, was declared at about 16:25 EDT due to the 
loss of offsite power for more than 15 minutes. 
Offsite power was restored to at least one safety 
bus at about 20:02 EDT on August 147The follow- 
ing equipment problems were noted: the service 
water to one of the emergency diesel generators 
developed a leak; a steam generator atmospheric 
dump valve did not control steam generator pres- 
sure in automatic and had to be shifted to manual; 
a steam trap associated with the turbine-driven 
AFW pump failed open, resulting in operators 
securing the turbine after 2.5 hours; loss of instru- 
ment air required operators to take manual control 
of charging and a letdown isolation occurred; and 
operators in the field could not use radios. The 
main generator was reconnected to the grid at 
about 12:58 EDT on August 17. 

Indian Point 3. Indian Point 3 is located 24 miles 
north of New York City on the Hudson River. It 
was generating about 1,010 MWe before the event. 
The reactor tripped due to loss of a reactor coolant 
pump that tripped because the auxiliary bus 
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frequency fluctuations actuated the under- 
frequency relay, which protects against inade- 
quate coolant flow through the reactor core. This 
reactor protection signal tripped the reactor, 
which resulted in turbine and generator trips. 

The auxiliary bus experienced the under- 
frequency due to fluctuating grid conditions. 
Offsite power was lost to all the plant auxiliary 
buses. The safety buses were reenergized from the 
emergency diesel generators. Decay heat removal 
systems maintained the cooling function for the 
reactor fuel. 

The lowest emergency declaration, an Unusual 
Event, was declared at about 16:23 EDT due to the 
loss of offsite power for more than 15 minutes. 
Offsite power was restored to at least one safety 
bus at about 20:12 EDT on August 14. The follow- 
ing equipment problems were noted: a steam gen- 
erator safety valve lifted below its desired setpoint 
and was gagged: loss of instrument air, including 
failure of the diesel backup compressor to start 
and failure of the backup nitrogen system, 
resulted in manual control of atmospheric dump 
valves and AFW pumps needing to be secured to 
prevent overfeeding the steam generators: a blown 
fuse in a battery charger resulted in a longer bat- 
tery discharge; a control rod drive mechanism 
cable splice failed, and there were high resistance 
readings on 345-kV breaker-1. These equipment 
problems required correction prior to start-up, 
which delayed the startup. The main generator 
was reconnected to the grid at about 05:03 EDT on 
August 22. 

Nine Mile 1. Nine Mile_l is located 6 miles north- 
east of Oswego, NY, in northern New York on Lake 
Ontario. It was generating about 600 MWe before 
the event. The reactor tripped in response to a tur- 
bine trip. The turbine tripped on light load protec- 
tion (which protects the turbine against a loss of 
electrical load), when responding to fluctuating 
grid conditions. The turbine trip caused fast clo- 
sure of the turbine valves, which, through acceler- 
ation relays on the control valves, create a signal to 
trip the reactor. After a time delay of 10 seconds, 
the generator tripped on reverse power. 

The safety buses were automatically deenergized 
due to low voltage and automatically reenergized 
from the emergency diesel generators. Decay heat 
removal systems maintained the cooling function 
for the reactor fuel. 

The lowest emergency declaration, an Unusual 
Event, was declared at about 16:33 EDT due to the 


degraded offsite power. Offsite power was 
restored to at least one safety bus at about 23:39 
EDT on August 14. The following additional 
equipment problems were noted: a feedwater 
block valve failed “as is” on the loss of voltage, 
resulting in a high reactor vessel level; fuses blew 
in fire circuits, causing control room ventilation 
isolation and fire panel alarms; and operators were 
delayed in placing shutdown cooling in service for 
several hours due to lack of procedure guidance to 
address particular plant conditions encountered 
during the shutdown. The main generator was 
reconnected to the grid at about 02:08 EDT on 
August 18. 

Nine Mile 2. Nine Mile 2 is located 6 miles north- 
east of Oswego, NY, in northern New York on Lake 
Ontario. It was generating about 1,193 MWe 
before the event. The reactor scrammed due to the 
actuation of pressure switches which detected low 
pressure in the hydraulic system that controls the 
turbine control valves. Low pressure in this sys- 
tem typically indicates a large load reject, for 
which a reactor trip is expected. In this case the 
pressure in the system was low because the con- 
trol system was rapidly manipulating the turbine 
control valves to control turbine speed, which was 
being affected by grid frequency fluctuations. 

After the reactor tripped, several reactor level con- 
trol valves did not reposition, and with the main 
feedwater system continuing to operate, a high 
water level in the reactor caused a turbine trip, 
which caused a generator trip. Offsite power was 
degraded but available to the plant auxiliary 
buses. The offsite power dropped below the nor- 
mal voltage levels, which- resulted in the safety 
buses being automatically energized from the 
emergency diesel generators. Decay heat removal 
systems maintained the cooling function for the 
reactor fuel. 

The lowest emergency declaration, an Unusual 
Event, was declared at about 17:00 EDT due to the 
loss of offsite power to the safety buses for more 
than 15 minutes. Offsite power was restored to at 
least one safety bus at about 01:33 EDT on August 
15. The following additional equipment problem 
was noted: a tap changer on one of the offsite 
power transformers failed, complicating the resto- 
ration of one division of offsite power. The main 
generator was reconnected to the grid at about 
19:34 EDT on August 17. 

Oyster Creek. Oyster Creek is located 9 miles 
south of Toms River, NJ, near the Atlantic Ocean. 
It was generating about 629 MWe before the event. 
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The reactor tripped due to a turbine trip. The tur- 
bine trip was the result of a generator trip due to 
actuation of a high Volts/Hz protective trip. The 
Volts/Hz trip is a generator/transformer protective 
feature. The plant safety and auxiliary buses trans- 
ferred from the main generator supply to the 
offsite power supply following the plant trip. 
Other than the plant transient, no equipment dr 
performance problems were determined to be 
directly related to the grid problems. 

Post-trip the operators did not get the mode switch 
to shutdown before main steam header pressure 
reached its isolation setpoint. The resulting MSIV 
closure complicated the operator's response 
because the normal steam path to the main con- 
denser was lost. The operators used the isolation 
condensers for decay heat removal. The plant 
safety and auxiliary buses remained energized 
from offsite power for the duration of the event, 
and the emergency diesel generators were not 
started. Decay heat removal systems maintained 
the cooling function for the reactor fuel. The main 
generator was reconnected to the grid at about 
05:02 EDT on August 17. 

Perry. Perry is located 7 miles northeast of Paines- 
ville, OH, in northern Ohio on Lake Erie. It was 
generating about 1,275 MWe before the event. The 
reactor tripped due to a turbine control valve fast 
closure trip signal. The turbine control valve fast 
closure trip signal was due to a generator under- 
frequency trip signal that tripped the generator 
and the turbine and was triggered by grid fre- 
quency fluctuations. Plant operators noted voltage 
fluctuations and spikes on the main transformer, 
and the Generator Out-of-Step Supervisory relay 
actuated approximately 30 minutes before the 
trip. This supervisory relay senses a ground fault 
on the grid. The purpose is to prevent a remote 
fault on the grid from causing a generator out-of- 
step relay to activate, which would result in a gen- 
erator trip. Approximately 30 seconds prior to the 
trip operators noted a number of spikes on the gen- 
erator field volt meter, which subsequently went 
offscale high. The MVAR and MW meters likewise 
went offscale high. 

The safety buses were deenergized and automati- 
cally reenergized from the emergency diesel gen- 
erators. Decay heat removal systems maintained 
the cooling function for the reactor fuel. The fol- 
lowing equipment problems were noted: a steam 
bypass valve opened; a reactor water clean-up sys- 
tem pump tripped; the off-gas system isolated, and 
a keep-fill pump was found to be air-bound, 


requiring venting and filling before the residual 
heat removal system loop A and the low pressure 
core spray system could be restored to service. 

The lowest emergency declaration, an Unusual 
Event, was declared at about 16:20 EDT due to the 
loss of offsite power. Offsite power was restored to 
at least one safety bus at about 18:13 EDT on 
August 14. The main generator was reconnected 
to the grid at about 23:15 EDT on August 21. After 
the plant restarted, a surveillance test indicated a 
problem with one emergency diesel generator. An 
NRC special inspection is in progress, reviewing 
emergency diesel generator performance and the 
keep-fill system. 

Nuclear Power Plants With a Significant 
Transient 

The electrical disturbance on August 14 had a sig- 
nificant impact on seven plants that continued to 
remain connected to the grid. For this review, sig- 
nificant impact means that these plants had signif- 
icant load adjustments that resulted in bypassing 
steam from the turbine generator, opening of relief 
valves, or requiring the onsite emergency diesel 
generators to automatically start due to low 
voltage. 

Nuclear Power Plants With a Non-Significant 
Transient 

Sixty-four nuclear power plants experienced 
non-significant transients caused by minor distur- 
bances on the electrical grid. These plants were 
able to respond to the disturbances through nor- 
mal control systems. Examples of these transients 
included changes in load of a few megawatts or 
changes in frequency of a few-tenths Hz. 

Nuclear Power Plants With No Transient 
Twenty-four nuclear power plants experienced no 
transient and saw essentially no disturbances on 
the grid, or were shut down at the time of the 
transient. 

General Observations Based on the Facts 
Found During Phase One 

The NWG has found no evidence that the shut- 
down of U.S. nuclear power plants triggered the 
outage or inappropriately contributed to its spread 
(i.e., to an extent beyond the normal tripping of 
the plants at expected conditions). This review did 
not identify any activity or equipment issues that 
appeared to start the transient on August 14, 2003. 
All nine plants that experienced a reactor trip 
were responding to grid conditions. The severity 
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of the transient caused generators, turbines, or 
reactor systems to reach a protective feature limit 
and actuate a plant shutdown. 

All nine plants tripped in response to those condi- 
tions in a manner consistent with the plant 
designs. All nine plants safely shut down. All 
safety functions were effectively accomplished, 
with few problems, and the plants were main- 
tained in a safe shutdown condition until their 
restart. Fermi 2, Nine Mile 1, Oyster Creek, and 
Perry tripped on turbine and generator protective 
features. Fitzpatrick, Ginna, Indian Point 2 and 3, 
and Nine Mile 2 tripped on reactor protective 
features. 

Nine plants used their emergency diesel genera- 
tors to power their safety-related buses during the 
power outage. Offsite power was restored to the 
safety buses after the grid was energized and the 
plant operators, in consultation with the transmis- 
sion system operators, decided the grid was stable. 
Although the Oyster Creek plant tripped, offsite 
power was never lost to their safety buses and the 
emergency diesel generators did not start and 
were not required. Another plant, Davis-Besse, 
was already shut down but lost power to the safety 
buses. The emergency diesel generators started 
and provided power to the safety buses as 
designed. 

For the eight remaining tripped plants and 
Davis-Besse (which was already shut down prior 
to the events of August 14), offsite power was 
restored to at least one safety bus after a period of 
time ranging from about 2 hours to about 14 hours, 
jwith an average time of about 7 hours. Although 
Ginna did not lose offsite power, the operators 
judged offsite power to be unstable and realigned 
the safety buses to the emergency diesel genera- 
tors. The second phase of the Power System Out- 
age Task Force will consider the implications of 
this in developing recommendations for future 
improvements. 

The licensees’ return to power operation follows a 
deliberate process controlled by plant procedures 
and NRC regulations. Ginna, Indian Point 2, Nine 
Mile 2, and Oyster Creek resumed electrical gener- 
ation on August 17. FitzPatrick and Nine Mile 1 
resumed electrical generation on August 18. Fermi 
2 resumed electrical generation on August 20. 
Perry resumed electrical generation on August 21. 
Indian Point 3 resumed electrical generation on 
August 22. Indian Point 3 had equipment issues 
(failed splices in the control rod drive mechanism 
power system) that required repair prior to restart. 


Ginna submitted a special request for enforcement 
discretion from the NRC to permit mode changes 
and restart with an inoperable auxiliary feedwater 
pump. The NRC granted the request for enforce- 
ment discretion. 

Findings of the Canadian Nuclear 
Working Group 

Summary 

On the afternoon of August 14, 2003, southern 
Ontario, along with the northeastern United 
States, experienced a widespread electrical power 
system outage. Eleven nuclear power plants in 
Ontario operating at high power levels at the time 
of the event either automatically shut down as a 
result of the grid disturbance or automatically 
reduced power while waiting for the grid to be 
reestablished. In addition, the Point Lepreau 
Nuclear Generating Station in New Brunswick 
was forced to reduce electricity production for a 
short period. 

The Canadian NWG was mandated to: review the 
sequence of events for each Canadian nuclear 
plant; determine whether any events caused or 
contributed to the power system outage; evaluate 
any potential safety issues arising as a result of the 
event; evaluate the effect on safety and the reli- 
ability of the grid of design features, operating pro- 
cedures, and regulatory requirements at Canadian 
nuclear power plants; and assess the impact of 
associated regulator performance and regulatory 
decisions. 

In Ontario, 11 nuclear units were operating and 
delivering power to the grid at the time of the grid 
disturbance: 4 at Bruce B, 4 at Darlington, and 3 at 
Pickering B. Of the 11 reactors, 7 shut down as a 
result of the event (1 at Bruce B, 3 at Darlington, 
and 3 at Pickering B). Four reactors (3 at Bruce B 
and 1 at Darlington) disconnected safely from the 
grid but were able to avoid shutting down and 
were available to supply power to the Ontario grid 
as soon as reconnection was enabled by Ontario’s 
Independent Market Operator (IMO). 

New Brunswick Power’s Point Lepreau Generating 
Station responded to the loss of grid event by cut- 
ting power to 460 MW, returning to fully stable 
conditions at 16:35 EDT, within 25 minutes of the 
event. Hydro Quebec's (HQ) grid was not affected 
by the power system outage, and HQ's Gentilly-2 
nuclear station continued to operate normally. 
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Having reviewed the operating data for each plant 
and the responses of the power stations and then- 
staff to the event, the Canadian NWG concludes 
the following: 

♦ None of the reactor operators had any advanced 
warning of impending collapse of the grid. 

> Trend data obtained indicate stable condi- 
tions until a few minutes before the event, 

»- There were no prior warnings from Ontario’s 
IMO. 

♦ Canadian nuclear power plants did not trigger 
the power system outage or contribute to its 
spread. Rather they responded, as anticipated, 
in order to protect equipment and systems from 
the grid disturbances. Plant data confirm the 
following. 

>• At Bruce B and Pickering B, frequency and/or 
voltage fluctuations on the grid resulted in 
the automatic disconnection of generators 
from the grid. For those units that were suc- 
cessful in maintaining the unit generators 
operational, reactor power was automatically 
reduced. 

> At Darlington, load swing on the grid led to 
the automatic reduction in power of the four 
reactors. The generators were, in turn, auto- 
matically disconnected from the grid. 

>- Three reactors at Bruce B and one at Darling- 
ton were returned to 60% power. These 
ractors were available to deliver power to the 
grid on the instructions of the IMO. 

> Three units at Darlington were placed in a 
zero-power hot state, and four units at 
Pickering B and one unit at Bruce B were 
placed in a guaranteed shutdown state. 

♦ There were no risks to health and safety of 
workers or the public as a result of the shut- 
down of the reactors. 

> Turbine, generator, and reactor automatic 
safety systems worked as designed to 
respond to the loss of grid. 

> Station operating staff and management fol- 
lowed approved Operating Policies & Princi- 
ples (OP&Ps) in responding to the loss of grid. 
At all times, operators and shift supervisors 
made appropriately conservative decisions in 
favor of protecting health and safety. 

The Canadian NWG commends the staff of 
Ontario Power Generation and Bruce Power for 
their response to the power system outage. At all 


times, staff acted in accordance with established 
OP&Ps, and took an appropriately conservative 
approach to decisions. 

During the course of its review, the NWG also 
identified the following secondary issues: 

♦ Equipment problems and design limitations at 
-Pickering B resulted in a temporary reduction in 
the effectiveness of some of the multiple safety 
barriers, although the equipment failure was 
within the unavailability targets found in the 
OP&Ps approved by the CNSC as part of Ontario 
Power Generation’s licence. 

♦ Existing OP&Ps place constraints on the use of 
adjuster rods to respond to events involving 
rapid reductions in reactor power. While 
greater flexibility with respect to use of adjuster 
rods would not have prevented the shutdown, 
some unit s, particularly those at Darlington, 
might have been able to return to service less 
than 1 hour after the initiating event. 

♦ Off-site power was unavailable for varying peri- 
ods of time, from approximately 3 hours at 
Bruce B to approximately 9 hours at Pickering 
A. Despite the high priority assigned by the IMO 
to restoring power to the nuclear stations, the 
stations had some difficulty in obtaining timely 
information about the status of grid recovery 
and the restoration of Class IV power. This 
information is important for Ontario Power 
Generation’s and Bruce Power's response 
strategy. 

♦ Required regulatory approvals from CNSC staff 
were obtained quickly and did not delay the 
restart of the units; however, CNSC staff was 
unable to immediately activate the CNSC's 
Emergency Operation Centre because of loss of 
power to the CNSC's head office building. 
CNSC staff, therefore, established communica- 
tions with licensees and the U.S. NRC from 
other locations. 

Introduction 

The primary focus of the Canadian NWG during 
Phase I was to address nuclear power plant 
response relevant to the power outage of August 
14, 2003. Data were collected from each power 
plant and analyzed in order to determine: the 
cause of the power outage; whether any activities 
at these plants caused or contributed to the power 
outage; and whether there were any significant 
safety issues. In order to obtain reliable and com- 
parable information and data from each nuclear 


❖ U.S.-Canada Power System Outage Task Force <=- Causes of the August 14th Blackout <> 


85 



298 


power plant, a questionnaire was developed to 
help pinpoint how each nuclear power plant 
responded to the August 14 grid transients. Where 
appropriate, additional information was obtained 
from the ESWG and SWG. 

The operating data from each plant were com- 
pared against the plant design specifications to 
determine whether the plants responded as 
expected. Based on initial plant responses to the 
questionnaire, supplemental questions were 
developed, as required, to further clarify outstand- 
ing matters. Supplementary information on the 
design features of Ontario's nuclear power plants 
was also provided by Ontario Power Generation 
and Bruce Power. The Canadian NWG also con- 
sulted a number of subject area specialists, includ- 
ing CNSC staff, to validate the responses to the 
questionnaire and to ensure consistency in their 
interpretation. 

Typical Design, Operational, and 
Protective Features of CANDU Nuclear 
Power Plants 

There are 22 CANDU nuclear power reactors in 
Canada — 20 located in Ontario at 5 multi-unit sta- 
tions (Pickering A and Pickering B located in 
Pickering, Darlington located in the Municipality 
of Clarington, and Bruce A and Bruce B located 
near Kincardine). There are also single-unit 
CANDU stations at Becancour, Quebec (Gentilly- 
2), and Point Lepreau, New Brunswick. 

In contrast to the pressurized water reactors used 
in the United States, which use enriched uranium 
fuel and a light water coolanhmoderator, all 
housed in a single, large pressure vessel, a CANDU 
reactor uses fuel fabricated from natural uranium, 
with heavy water as the coolant and moderator. 
The fuel and pressurized heavy water coolant are 
contained in 380 to 480 pressure tubes housed in a 
calandria containing the heavy water moderator 
under low pressure. Heat generated by the fuel is 
removed by heavy water coolant that flows 
through the pressure tubes and is then circulated 
to the boilers to produce steam from demineral- 
ized water. 

While the use of natural uranium fuel offers 
important benefits from the perspectives of safe- 
guards and operating economics, one drawback is 
that it restricts the ability of a CANDU reactor to 
recover from a large power reduction. In particu- 
lar, the lower reactivity of natural uranium fuel 
means that CANDU reactors are designed with a 


small number of control rods (called "adjuster 
rods”) that are only capable of accommodating 
power reductions to 60%. The consequence of a 
larger power reduction is that the reactor will "poi- 
son out” and cannot be made critical for up to 2 
days following a power reduction. By comparison, 
the use of enriched fuel enables a typical pressur- 
ized water reactor to operate with a large number 
of control rods that can be withdrawn to accom- 
modate power reductions to zero power. 

A unique feature of some CANDU plants — 
namely, Bruce B and Darlington — is a capability to 
maintain the reactor at 60% full power if the gen- 
erator becomes disconnected from the grid and to 
maintain this "readiness” condition if necessary 
for days. Once reconnected to the grid, the unit 
can be loaded to 60% full power within several 
minutes and can achieve full power within 24 
hours. 

As with other nuclear reactors, CANDU reactors 
normally operate continuously at full power 
except when shut down for maintenance and 
inspections. As such, while they provide a stable 
source of baseload power generation, they cannot 
provide significant additional power in response 
to sudden increases in demand. CANDU power 
plants are not designed for black-start operation; 
that is, they are not designed to start up in the 
absence of power from the grid. 

Electrical Distribution Systems 

The electrical distribution systems at nuclear 
power plants are designed to satisfy the high 
safety and reliability requirements for nuclear sys- 
tems. This is achieved through-flexible bus 
arrangements, high capacity standby power gener- 
ation, and ample redundancy in equipment. 

Where continuous power is required, power is 
supplied either from batteries (for continuous DC 
power, Class I) or via inverters (for continuous AC 
power, Class II). AC supply for safety-related 
equipment, which can withstand short interrup- 
tion (on the order of 5 minutes), is provided by 
Class III power. Class III power is nominally sup- 
plied through Class IV; when Class IV becomes 
unavailable, standby generators axe started auto- 
matically, and the safety-related loads are picked 
up within 5 minutes of the loss of Class IV power. 

The Class IV power is an AC supply to reactor 
equipment and systems that can withstand longer 
interruptions in power. Class IV power can be sup- 
plied either from the generator through a 
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transformer or from the grid by another trans- 
former. Class IV power is not required for reactors 
to shut down safely. 

In addition to the four classes of power described 
above, there is an additional source of power 
known as the Emergency Power System (EPS). 
EPS is a separate power system consisting of its 
own on-site power generation and AC and DC dis- 
tribution systems whose normal supply is from 
the Class III power system. The purpose of the EPS 
system is to provide power to selected safety- 
related loads following common mode incidents, 
such as seismic events. 

Protective Features of CANDU Nuclear Power 
Plants 

CANDU reactors typically have two separate, 
independent and diverse systems to shut down 
the reactor in the event of an accident or transients 
in the grid. Shutdown System 1 (SDSl) consists of 
a large number of cadmium rods that drop into the 
core to decrease the power level by absorbing neu- 
trons. Shutdown System 2 (SDS2) consists of 
high-pressure injection of gadolinium nitrate into 
the low-pressure moderator to decrease the power 
level by absorbing neutrons. Although Pickering A 
does not have a fully independent SDS2, it does 
have a second shutdown mechanism, namely, the 
fast drain of the moderator out of the calandria; 
removal of the moderator significantly reduces the 
rate of nuclear fission, which reduces reactor 
power. Also, additional trip circuits and shutoff 
rods have recently been added to Pickering A Unit 
4 (Shutdown System Enhancement, or SDS-E). 
Both SDSl and SDS2 are capable of reducing reac- 
tor power from 100%~to about 2% within a few 
seconds of trip initiation. 

Fuel Heat Removal Features of CANDU 
Nuclear Power Plants 

Following the loss of Class IV power and shut- 
down of the reactor through action of SDSl and/or 
SDS2, significant heat will continue to be gener- 
ated in the reactor fuel from the decay of fission 
products. The CANDU design philosophy is to 
provide defense in depth in the heat removal 
systems. 

Immediately following the trip and prior to resto- 
ration of Class III power, heat will be removed 
from the reactor core by natural circulation of 
coolant through the Heat Transport System main 
circuit following rundown of the main Heat Trans- 
port pumps (first by thermosyphoning and later by 
intermittent buoyancy induced flow). Heat will be 


rejected from the secondary side of the steam gen- 
erators through the atmospheric steam discharge 
valves. This mode of operation can be sustained 
for many days with additional feedwater supplied 
to the steam generators via the Class III powered 
auxiliary steam generator feed pump(s). 

In the event that the auxiliary feedwater system 
becomes unavailable, there are two alternate EPS 
powered water supplies to steam generators, 
namely, the Steam Generator Emergency Coolant 
System and the Emergency Service Water System. 
Finally, a separate and independent means of 
cooling the fuel is by forced circulation by means 
of the Class III powered shutdown cooling system; 
heat removal to the shutdown cooling heat 
exchangers is by means of the Class III powered 
components of die Service Water System. 

CANDU Reactor Response to 
Loss-of-Grid Event 

Response to Loss of Grid 

In the event of disconnection from the grid, power 
to safely shut down the reactor and maintain 
essential systems will be supplied from batteries 
and standby generators. The specific response of a 
reactor to disconnection from the grid will depend 
on the reactor design and the condition of the unit 
at the time of the event. 

60% Reactor Power: All CANDU reactors are 
designed to operate at 60% of full power following 
the loss of off-site power. They can operate at this 
level as long as demineralized water is available 
for the boilers. At Darlington and Bruce B, steam 
can be diverted to the condensers and recirculated 
to the boilers. At Pickering A and Pickering B, 
excess steam is vented to the atmosphere, thereby 
limiting the operating time to the available inven- 
tory of demineralized water. 

0% Reactor Power, Hot: The successful transition 
from 100% to 60% power depends on several sys- 
tems responding properly, and continued opera- 
tion is not guaranteed. The reactor may shut down 
automatically through the operation of the process 
control systems or through the action of either of 
the shutdown systems. 

Should a reactor shutdown occur following a load 
rejection, both Class IV power supplies (from the 
generator and the grid) to that unit will become 
unavailable. The main Heat Transport pumps 
will trip, leading to a loss of forced circulation of 
coolant through the core. Decay heat will be con- 
tinuously removed through natural circulation 
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(thermosyphoning) to the boilers, and steam pro- 
duced in the boilers will be exhausted to the 
atmosphere via atmospheric steam discharge 
valves. The Heat Transport System will be main- 
tained at around 250 to 265 degrees Celsius during 
thermosyphoning. Standby generators will start 
automatically and restore Class III power to key 
safety -related systems. Forced circulation in the 
Heat Transport System will be restored once 
either Class III or Class IV power is available. 

When shut down, the natural decay of fission 
products will lead to the temporary buildup of 
neutron absorbing elements in the fuel. If the reac- 
tor is not quickly restarted to reverse this natural 
process, it will “poison-out.” Once poisoned-out, 
the reactor cannot return to operation until the fis- 
sion products have further decayed, a process 
which typically takes up to 2 days. 

Overpoisoned Guaranteed Shutdown State: In 
the event that certain problems are identified 
when reviewing the state of the reactor after a sig- 
nificant transient, the operating staff will cool 
down and depressurize the reactor, then place it in 
an overpoisoned guaranteed shutdown state (GSS) 
through the dissolution of gadolinium nitrate into 
the moderator. Maintenance will then be initiated 
to correct the problem. 

Return to Service Following Loss of Grid 

The return to service of a unit following any one of 
the above responses to a loss-of-grid event is dis- 
cussed below. It is important to note that the 
descriptions provided relate to operations on a 
single unit. At multi-unit stations, the return to 
service of-several units cannot always proceed in 
parallel, due to constraints on labor availability 
and the need to focus on critical evolutions, such 
as taking the reactor from a subcritical to a critical 
state. 

60% Reactor Power: In this state, the unit can be 
resynchronized consistent with system demand, 
and power can be increased gradually to full 
power over approximately 24 hours. 

0% Reactor Power, Hot: In this state, after approx- 
imately 2 days for the poison-out, the turbine can 
be run up and the unit synchronized. The reactor 
may shut down automatically through the opera- 
tion of the process control systems or through the 
action of either of the shutdown systems. Thereaf- 
ter, power can be increased to high power over the 
next day. This restart timeline does not include 
the time required for any repairs or maintenance 
that might have been necessary during the outage. 


Overpoisoned Guaranteed Shutdown State: Plac- 
ing the reactor in a GSS after it has been shut down 
requires approximately 2 days. Once the condi- 
tion that required entry to the GSS is rectified, the 
restart requires removal of the guarantee, removal 
of the gadolinium nitrate through ion exchange 
process, heatup of the Heat Transport System, and 
finally synchronization to the grid. Approximately 
4 days are required to complete these restart activ- 
ities. In total, 6 days from shutdown are required 
to return a unit to service from the GSS, and this 
excludes any repairs that might have been 
required while in the GSS. 

Summary of Canadian Nuclear Power 
Plant Response to and Safety During the 
August 14 Outage 

On the afternoon of August 14, 2003, 15 Canadian 
nuclear units were operating: 13 in Ontario, 1 in 
Quebec, and 1 in New Brunswick. Of the 13 
Ontario reactors that were critical at the time of 
the event, 11 were operating at or near full power 
and 2 at low power (Pickering B Unit 7 and 
Pickering A Unit 4). All 13 of the Ontario reactors 
disconnected from the grid as a result of the grid 
disturbance. Seven of the 11 reactors operating at 
high power shut down, while the remaining 4 
operated in a planned manner that enabled them 
to remain available to reconnect to the grid at the 
request of Ontario’s IMO. Of the 2 Ontario reactors 
operating at low power, Pickering A Unit 4 tripped 
automatically, and Pickering B Unit 7 was tripped 
manually and shut down. In addition, a transient 
was experienced at New Brunswick Power’s Point 
Lepreau Nuclear Generating Station, resulting in a 
reduction in power. Hydro Quebec’s Gentilly-2 
nuclear station continued to operate normally as 
the Hydro Quebec grid was not affected by the grid 
disturbance. 

Nuclear Power Plants With Significant 
Transients 

Pickering Nuclear Generating Station. The 
Pickering Nuclear Generating Station (PNGS) is 
located in Pickering, Ontario, on the shores of 
Lake Ontario, 30 kilometers east of Toronto. It 
houses 8 nuclear reactors, each capable of deliver- 
ing 515 MW to the grid. Three of the 4 units at 
Pickering A (Units 1 through 3) have been shut 
down since late 1997. Unit 4 was restarted earlier 
this year following a major refurbishment and was 
in the process of being commissioned at the time 
of the event. At Pickering B, 3 units were operating 
at or near 100% prior to the event, and Unit 7 was 
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being started up following a planned maintenance 
outage. 

Pickering A As part of the commissioning process, 
Unit 4 at Pickering A was operating at 12% power 
in preparation for synchronization to the grid. The 
reactor automatically tripped on SDSl due to Heat 
Transport Low Coolant Flow, when the Heat 
Transport main circulating pumps ran down fol- 
lowing the Class IV power loss. The decision was 
then made to return Unit 4 to the guaranteed shut- 
down state. Unit 4 was synchronized to the grid on 
August 20, 2003. Units 1, 2 and 3 were in lay-up 
mode. 

Pickering B. The Unit 5 Generator Excitation Sys- 
tem transferred to manual control due to large 
voltage oscillations on the grid at 16:10 EDT and 
then tripped on Loss of Excitation about 1 second 
later (prior to grid frequency collapse). In response 
to the generator trip, Class IV buses transferred to 
the system transformer and the reactor setback. 
The grid frequency collapse caused the System 
Service Transformer to disconnect from the grid, 
resulting in a total loss of Class IV power. The 
reactor consequently tripped on the SDSl Low 
Gross Flow parameter followed by an SDS2 trip 
due to Low Core Differential Pressure. 

The Unit 6 Generator Excitation System also 
transferred to manual control at 16:10 EDT due to 
large voltage oscillations on the grid and the gen- 
erator remained connected to the grid in manual 
voltage control. Approximately 65 seconds into 
the event, the grid under-frequency caused all the 
Class IV buses to transfer to the Generator Service 
Transformer. Ten seconds later, the generator sep- 
arated from the Grid. Five seconds later, the gener- 
ator tripped on Loss of Excitation, which caused a 
total loss of Class IV power. The reactor conse- 
quently tripped on the SDSl Low Gross Flow 
parameter, followed by an SDS2 trip due to Low 
Core Differential Pressure. 

Unit 7 was coming back from a planned mainte- 
nance outage and was at 0.9% power at the time of 
the event. The unit was manually tripped after 
loss of Class IV power, in accordance with proce- 
dures and returned to guaranteed shutdown state. 

Unit 8 reactor automatically set back on load rejec- 
tion. The setback would normally have been ter- 
minated at 20% power but continued to 2% power 
because of the low boiler levels. The unit subse- 
quently tripped on the SDSl Low Boiler Feedline 
Pressure parameter due to a power mismatch 
between the reactor and the turbine. 


The following equipment problems were noted. At 
Pickering, the High Pressure Emergency Coolant 
Injection System (HPECIS) pumps are designed to 
operate from a Class IV power supply. As a result 
of the shutdown of all the operating units, the 
HPECIS at both Pickering A and Pickering B 
became unavailable for 5.5 hours. (The operating 
licenses for Pickering A and Pickering B permit 
the HPECIS to be unavailable for up to 8 hours 
annually. This was the first unavailability of the 
year.) In addition, Emergency High Pressure Ser- 
vice Water System restoration for all Pickering B 
units was delayed because of low suction pressure 
supplying the Emergency High Pressure Service 
Water pumps. Manual operator intervention was 
required to restore some pumps back to service. 

Units were synchronized to the grid as follows: 
Unit 8 on August 22, Unit 5 on August 23, Unit 6 
on August 25, and Unit 7 on August 29. 

Darlington Nuclear Generating Station. Four 
reactors are located at the Darlington Nuclear Gen- 
eration Station, which is on the shores of Lake 
Ontario in the Municipality of Clarington, 70 kilo- 
meters east of Toronto. All four of the reactors are 
licensed to operate at 100% of full power, and 
each is capable of delivering approximately 880 
MW to the grid. 

Unit 1 automatically stepped back to the 60% 
reactor power state upon load rejection at 16:12 
EDT. Approval by the shift supervisor to automati- 
cally withdraw the adjuster rods could not be pro- 
vided due to the brief period of time for the shift 
supervisor to complete the verification of systems 
as per procedure. The decreasing steam pressure 
and turbine frequency then required the reactor to 
be manually tripped on SDSl , as per procedure for 
loss of Class IV power. The trip occurred at 16:24 
EDT, followed by a manual turbine trip due to 
under-frequency concerns. 

Like Unit 1, Unit 2 automatically stepped back 
upon load rejection at 16:12 EDT. As with Unit 1, 
there was insufficient time for the shift supervisor 
to complete the verification of systems, and faced 
with decreasing steam pressure and turbine fre- 
quency, the decision was made to shut down Unit 
2. Due to under-frequency on the main Primary 
Heat Transport pumps, the turbine was tripped 
manually which resulted in an SDSl trip at 16:28 
EDT. 

Unit 3 experienced a load rejection at 16:12 EDT, 
and during the stepback Unit 3 was able to sustain 
operation with steam directed to the condensers. 
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After system verifications were complete, approv- 
al to place the adjuster rods on automatic was 
obtained in time to recover, at 59% reactor power. 

The unit was available to resynchronize to the 
grid. Unit 4 experienced a load rejection at 16:12 
EDT, and required a manual SDSl trip due to the 
loss of Class II bus. This was followed by a manual 
turbine trip. 

The following equipment problems were noted: 
Unit 4 Class II inverter trip on BUS A3 and subse- 
quent loss of critical loads prevented unit recov- 
ery. The Unit 0 Emergency Power System BUS 
B135 power was lost until the Class III power was 
restored. (A planned battery bank B135 change 
out was in progress at the time of the blackout.) 

Units were synchronized to the grid as follows: 
Unit 3 at 22:00 EDT on August 14; Unit 2 on 
August 17, 2003; Unit 1 on August 18, 2003; and 
Unit 4 on August 18, 2003. 

Bruce Power. Eight reactors are located at Bruce 
Power on the eastern shore of Lake Huron between 
Kincardine and Port Elgin, Ontario. Units 5 
through 8 are capable of generating 840 MW each. 
Presently these reactors are operating at 90% of 
full power due to license conditions imposed by 
the CNSC. Units 1 through 4 have been shutdown 
since December 31, 1997. Units 3 and 4 are in the 
process of startup. 

Bruce A. Although these reactors were in guaran- 
teed shutdown state, they were manually tripped, 
in accordance with operating procedures. SDSl 
was manually tripped on Units 3 and 4, as per pro- 
cedures for a loss of Class IV power event. SDSl 
was re-poised on both units when the station 
power supplies were stabilized. The emergency 
transfer system functioned as per design, with the 
Class III standby generators picking up station 
electrical loads. The recently installed Qualified 
Diesel Generators received a start signal and were 
available to pick up emergency loads if necessary. 

Bruce B. Units 5, 6, 7, and 8 experienced initial 
generation rejection and accompanying stepback 
on all four reactor units. All generators separated 
from the grid on under-frequency at 16:12 EDT. 
Units 5, 7, and 8 maintained reactor power at 60% 
of full power and were immediately available for 
reconnection to the grid. 

Although initially surviving the loss of grid event. 
Unit 6 experienced an SDSl trip on insufficient 
Neutron Over Power (NOP) margin. This occurred 


while withdrawing Bank 3 of the adjusters in an 
attempt to offset the xenon transient, resulting in a 
loss of Class IV power. 

The following equipment problems were noted: 
An adjuster rod on Unit 6 had been identified on 
August 13, 2003, as not working correctly. Unit 6 
experienced a High Pressure Recirculation Water 
line leak, and the Closed Loop Demineralized 
Water loop lost inventory to the Emergency Water 
Supply System. 

Units were synchronized to the grid as follows: 
Unit 8 at 19:14 EDT on August 14, 2003; Unit 5 at 
21:04 EDT on August 14; and Unit 7 at 21:14 EDT 
on August 14, 2003. Unit 6 was resynchronized at 
02:03 EDT on August 23, 2003, after maintenance 
was conducted. 

Point Lepreau Nuclear Generating Station. The 
Point Lepreau nuclear station overlooks the Bay of 
Fundy on the Lepreau Peninsula, 40 kilometers 
southwest of Saint John, New Brunswick. Point 
Lepreau is a single-unit CANDU 6, designed for a 
gross output of 680 MW. It is owned and operated 
by New Brunswick Power. 

Point Lepreau was operating at 91.5% of full 
power (610 MWe) at the time of the event. When 
the event occurred, the unit responded to changes 
in grid frequency as per design. The net impact 
was a short-term drop in output by 140 MW, with 
reactor power remaining constant and excess ther- 
mal energy being discharged via the unit steam 
discharge valves. During the 25 seconds of the 
event, theunit stabilizer operated numerous times 
to help dampen the turbine generator speed oscil- 
lations that were being introduced by the grid fre- 
quency changes. Within 25 minutes of the event 
initiation, the turbine generator was reloaded to 
610 MW. Given the nature of the event that 
occurred, there were no unexpected observations 
on the New Brunswick Power grid or at Point 
Lepreau Generating Station throughout the ensu- 
ing transient. 

Nuclear Power Plants With No Transient 

Gentilly-2 Nuclear Station. Hydro Quebec owns 
and operates Gentilly-2 nuclear station, located on 
the south shore of the St. Lawrence River opposite 
the city of Trois-RiviSres, Quebec. Gentilly-2 is 
capable of delivering approximately 675 MW to 
Hydro Quebec's grid. The Hydro Quebec grid was 
not affected by the power system outage and 
Gentilly-2 continued to operate normally. 
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General Observations Based on the Facts 

Found During Phase One 

Following the review of the data provided by the 

Canadian nuclear power plants, the Nuclear 

Working Group concludes the following: 

♦ None of the reactor operators had any advanced 
warning of impending collapse of the grid. 

♦ Canadian nuclear power plants did not trigger 
the power system outage or contribute to its 
spread. 

♦ There were no risks to the health and safety of 
workers or the public as a result of the concur- 
rent shutdown of several reactors. Automatic 
safety systems for the turbine generators and 
reactors worked as designed. (See Table 7.2 for 
a summary of shutdown events for Canadian 
nuclear power plants.) 

The NWG also identified the following secondary 

issues: 

♦ Equipment problems and design limitations at 
Pickering B resulted in a temporary reduction in 
the effectiveness of some of the multiple safety 
barriers, although the equipment failure was 
within the unavailability targets found in the 
OP&Ps approved by the CNSC as part of Ontario 
Power Generation's license. 


♦ Existing OP&Ps place constraints on the use of 
adjuster rods to respond to events involving 
rapid reductions in reactor power. While 
greater flexibility with respect to use of adjuster 
rods would not have prevented the shutdown, 
some units, particularly those at Darlington, 
might have been able to return to service less 
than 1 hour after the initiating event. 

♦ Off-site power was unavailable for varying peri- 
ods of time, from approximately 3 hours at 
Bruce B to approximately 9 hours at Pickering 
A. Despite the high priority assigned by the IMO 
to restoring power to the nuclear stations, the 
stations had some difficulty obtaining timely 
information about the status of grid recovery 
and the restoration of Class IV power. This 
information is important for Ontario Power 
Generation’s and Bruce Power's response 
strategy. 

♦ Required regulatory approvals from CNSC staff 
were obtained quickly and did not delay the 
restart of the units; however, CNSC staff was 
unable to immediately activate the CNSC’s 
Emergency Operation Centre because of loss of 
power to the CNSC’s head office building. 
CNSC staff, therefore, established communica- 
tions with licensees and the U.S. NRC from 
other locations. 
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Table 7.2. Summary of Shutdown Events for Canadian Nuclear Power Plants 




Operating Status at Time of 
Event 

Response to Event 
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V 
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a Pickering A Unit 1 tripped as a result of electrical bus configuration immediately prior to the event which resulted in a temporary 
loss of Class II power, 
dickering A Unit 4 also tripped on SDS-E. 

Notes: Unit 7 at Pickering B was operating at low power, warming up prior to reconnecting to the grid after a maintenance outage. 
Unit 4 at Pickering A was producing at low power, as part of the reactor's commissioning after extensive refurbishment since being 
shut down in 1997. 
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8. Physical and Cyber Security Aspects of the Blackout 


Summary 

The objective of the Security Working Group 
(SWG) is to determine what role, if any, that a 
malicious cyber event may have played in caus- 
ing, or contributing to, the power outage of August 
14, 2003, Analysis to date provides no evidence 
that malicious actors are responsible for, or con- 
tributed to, the outage. The SWG acknowledges 
reports of al-Qaeda claims of responsibility for the 
power outage of August 14, 2003; however, those 
claims are not consistent with the SWG's findings 
to date. There is also no evidence, nor is there any 
information suggesting, that viruses and worms 
prevalent across the Internet at the time of the out- 
age had any significant impact on power genera- 
tion and delivery systems. SWG analysis to date 
has brought to light certain concerns with respect 
to: the possible failure of alarm software; links to 
control and data acquisition software; and the lack 
of a system or process for some operators to view 
adequately the status of electric systems outside 
their immediate control. 

Further data collection and analysis will be under- 
taken by the SWG to test the findings detailed in 
this interim report and to examine more fully the 
cyber security aspects of the power outage. The 
outcome of Electric System Working Group 
(ESWG) root cause analysis will serve to focus this 
work. As the significant cyber events are identi- 
fied by the ESWG, the SWG will examine them 
from a security perspective. 

Security Working Group: 
Mandate and Scope 

It is widely recognized that the increased reliance 
on information technology (IT) by critical infra- 
structure sectors, including the energy sector, has 
increased their vulnerability to disruption via 
cyber means. The ability to exploit these vulnera- 
bilities has been demonstrated in North America. 
The SWG was established to address the 
cyber-related aspects of the August 14, 2003, 
power outage. The SWG is made up of U.S. and 


Canadian Federal, State, Provincial, and local 
experts in both physical and cyber security. For 
the purposes of its work, the SWG has defined a 
“malicious cyber event” as the manipulation of 
data, software or hardware for the purpose of 
deliberately disrupting the systems that control 
and support the generation and delivery of electric 
power. 

The SWG is working closely with the U.S. and 
Canadian law enforcement, intelligence, and 
homeland security communities to examine the 
possible role of malicious actors in the power out- 
age of August 14, 2003. A primary activity to date 
has been the collection and review of available 
intelligence that may relate to the outage. 

The SWG is also collaborating with the energy 
industry to examine the cyber systems that control 
power generation and delivery operations, the 
physical security of cyber assets, cyber policies 
and procedures, and the functionality of support- 
ing infrastructures-such as communication sys- 
tems and backup power generation, which 
facilitate the smooth-running operation of cyber 
assets-to determine whether the operation of these 
systems was affected by malicious activity. The 
collection of information alongrthese avenues of 
inquiry is ongoing. 

The SWG is coordinating its efforts with those of 
the other Working Groups, and there is a signifi- 
cant interdependence on the work products and 
findings of each group. The SWG’s initial focus is 
on the cyber operations of those companies in the 
United States involved in the early stages of the 
power outage timeline, as identified by the ESWG. 
The outcome of ESWG analysis will serve to iden- 
tify key events that may have caused, or contrib- 
uted to, the outage. As the significant cyber events 
are identified, the SWG will examine them from a 
security perspective. The amount of information 
for analysis is identified by the ESWG as pertinent 
to the SWG’s analysis is considerable. 

Examination of the physical, non-cyber infrastruc- 
ture aspects of the power outage of August 14, 
2003, is outside the scope of the SWG’s analysis. 
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Nevertheless, if a breach of physical security unre- 
lated to the cyber dimensions of the infrastructure 
comes to the SWG’s attention during the course of 
the work of the Task Force, the SWG will conduct 
the necessary analysis. 

Also outside the scope of the SWG's work is analy- 
sis of the cascading impacts of the power outage 
on other critical infrastructure sectors. Both the 
Canadian Office of Critical Infrastructure Protec- 
tion and Emergency Preparedness (OCIPEP) and 
the U.S, Department of Homeland Security (DHS) 
are examining these issues, but not within the 
context of the Task Force. The SWG is closely 
coordinating its efforts with OCIPEP and DHS. 

Cyber Security 
in the Electricity Sector 

The generation and delivery of electricity has 
been, and continues to be, a target of malicious 
groups and individuals intent on disrupting the 
electric power system. Even attacks that do not 
directly target the electricity sector can have dis- 
ruptive effects on electricity system operations. 
Many malicious code attacks, by their very nature, 
are unbiased and tend to interfere with operations 
supported by vulnerable applications. One such 
incident occurred in January 2003, when the 
“Slammer” Internet worm took down monitoring 
computers at FirstEnergy Corporation’s idled 
Davis-Besse nuclear plant. A subsequent report by 
the North American Electric Reliability Council 
(NERC) concluded that, although it caused no out- 
ages, the infection blocked commands that oper- 
ated other power utilities. The report, “NRC Issues 
Information Notice on Potential of Nuclear Power 
Plant Network to Worm Infection,” is available at 
web site http://www.nrc.gov/reading-irn/doc- 
collections/news/2003/03-1 08.html. 

This example, among others, highlights the 
increased vulnerability to disruption via cyber 
means faced by North America's critical infra- 
structure sectors, including the energy sector. Of 
specific concern to the U.S. and Canadian govern- 
ments are the Supervisory Control and Data 
Acquisition (SCADA) systems, which contain 
computers and applications that perform a wide 
variety of functions across many industries. In 
electric power, SCADA includes telemetry for sta- 
tus and control, as well as Energy Management 
Systems (EMS), protective relaying, and auto- 
matic generation control. SCADA systems were 


developed to maximize functionality and 
interoperability, with little attention given to 
cyber security. These systems, many of which 
were intended to be isolated, are now, for a variety 
of business and operational reasons, either 
directly or indirectly connected to the global 
Internet. For example, in some instances, there 
may be a need for employees to monitor SCADA 
systems remotely. However, connecting SCADA 
systems to a remotely accessible computer net- 
work can present security risks. These risks 
include the compromise of sensitive operating 
information and the threat of unauthorized access 
to SCADA systems’ control mechanisms. 

Security has always been a priority for the electric- 
ity sector in North America; however, it is a 
greater priority now than ever before. Electric sys- 
tem operators recognize that the threat environ- 
ment is changing and that the risks are greater 
than in the past, and they have taken steps to 
improve their security postures. NERC’s Critical 
Infrastructure Protection Advisory Group has 
been examining ways to improve both the physi- 
cal and cyber security dimensions of the North 
American power grid. This group includes Cana- 
dian and U.S. industry experts in the areas of 
cyber security, physical security and operational 
security. The creation of a national SCADA pro- 
gram to improve the physical and cyber security of 
these control systems is now also under discus- 
sion in the United States. The Canadian Electrical 
Association Critical Infrastructure Working Group 
is examining similar measures. 

Information Collection 
and Analysis 

In addition to analyzing information already 
obtained from stakeholder interviews, telephone 
transcripts, law enforcement and intelligence 
information, and other ESWG working docu- 
ments, the SWG will seek to review and analyze 
other sources of data on the cyber operations of 
those companies in the United States involved in 
the early stages of the power outage timeline, as 
identified by the ESWG. Available information 
includes log data from routers, intrusion detection 
systems, firewalls, and EMS; change management 
logs; and physical security materials. Data are cur- 
rently being collected, in collaboration with the 
private sector and with consideration toward its 
protection from further disclosure where there are 
proprietary or national security concerns. 
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The SWG is divided into six sub-teams to address 
the discrete components of this investigation: 
Cyber Analysis, Intelligence Analysis, Physical 
Analysis, Policies and Procedures, Supporting 
Infrastructure, and Root Cause Liaison. The SWG 
organized itself in this manner to create a holistic 
approach to each of the main areas of concern 
with regard to power grid vulnerabilities. Rather 
than analyze each area of concern separately, the 
SWG sub-team structure provides a more compre- 
hensive framework in which to investigate 
whether malicious activity was a cause of the 
power outage of August 14, 2003. Each sub-team is 
staffed with Subject Matter Experts (SMEs) from 
government, industry, and academia to provide 
the analytical breadth and depth necessary to 
complete its objective. A detailed overview of the 
sub-team structure and activities, those planned 
and those taken, for each sub-team is provided 
below. 

Cyber Analysis 

The Cyber Analysis sub-team is led by the CERT® 
Coordination Center (CERT/CC) at Carnegie 
Mellon University and the Royal Canadian 
Mounted Police (RCMP). This team is focused on 
analyzing and reviewing the electronic media of 
computer networks in which online communica- 
tions take place. The sub-team is examining these 
networks to determine whether they were mali- 
ciously used to cause, or contribute to, the August 
14 outage. It is specifically reviewing the existing 
cyber topology, cyber logs, and EMS logs. The 
team is also conducting interviews with vendors 
to identify known system flaws and vulnerabili- 
ties. The sub-team is collecting, processing, and 
synthesizing data to determine whether a mali- 
cious cyber-related attack was a direct or indirect 
cause of the outage. 

This sub-team has taken a number of steps in 
recent weeks, including reviewing NERC reliabil- 
ity standards to gain a better understanding of the 
overall security posture of the electric power 
industry. Additionally, the sub-team participated 
in meetings in Baltimore on August 22 and 23, 
2003. The meetings provided an opportunity for 
the cyber experts and the power industry experts 
to understand the details necessary to conduct an 
investigation. The cyber data retention request 
was produced during this meeting. 

Members of the sub-team also participated in the 
NERC/Department of Energy (DOE) Fact Finding 
meeting held in Newark, New Jersey, on Septem- 
ber 8, 2003. Each company involved in the outage 


provided answers to a set of questions related to 
the outage. The meeting helped to provide a better 
understanding of what each company experi- 
enced before, during, and after the outage. Addi- 
tionally, sub-team members participated in 
interviews with the control room operators from 
FirstEnergy on October 8 and 9, 2003, and from 
Cinergy on October 10, 2003. These interviews 
have identified several key areas for further 
discussion. 

The Cyber Analysis sub-team continues to gain a 
better understanding of events on August 14, 
2003. Future analysis will be driven by informa- 
tion received from the ESWG’s Root Cause Analy- 
sis sub-team and will focus on: 

♦ Conducting additional interviews with control 
room operators and IT staff from the key compa- 
nies involved in the outage. 

♦ Conducting interviews with the operators and 
IT staff responsible for the NERC Interchange 
Distribution Calculator system. Some reports 
indicate that this system may have been 
unavailable during the time of the outage. 

♦ Conducting interviews with key vendors for the 
EMS. 

♦ Analyzing the configurations of routers, fire- 
walls, intrusion detection systems, and other 
network devices to get a better understanding of 
potential weaknesses in the control system 
cyber defenses. 

♦ Analyzing logs and other information for signs 
of unauthorized activity. 

Intelligence Analysis — 

The Intelligence Analysis sub-team is led by DHS 
and the RCMP, which are working closely with 
Federal, State, and local law enforcement, intelli- 
gence, and homeland security organizations to 
assess whether the power outage was the result of 
a malicious attack. Preliminary analysis provides 
no evidence that malicious actors-either individu- 
als or organizations -are responsible for, or contrib- 
uted to, the power outage of August 14, 2003. 
Additionally, the sub-team has found no indica- 
tion of deliberate physical damage to power gener- 
ating stations and delivery lines on the day of the 
outage, and there are no reports indicating that the 
power outage was caused by a computer network 
attack. 

Both U.S. and Canadian government authorities 
provide threat intelligence information to their 
respective energy sectors when appropriate. No 
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intelligence reports before, during, or after the 
power outage indicated any specific terrorist plans 
or operations against the energy infrastructure. 
There was, however, threat information of a gen- 
eral nature relating to the sector, which was pro- 
vided to the North American energy industry by 
U.S. and Canadian government agencies in late 
July 2003. This information indicated that 
al-Qaeda might attempt to carry out a physical 
attack involving explosions at oil production facil- 
ities, power plants, or nuclear plants on the U.S. 
East Coast during the summer of 2003. The type of 
physical attack described in the intelligence that 
prompted this threat warning is not consistent 
with the events of the power outage; there is no 
indication of a kinetic event before, during, or 
immediately after the August 14 outage. 

Despite all the above indications that no terrorist 
activity caused the power outage, al-Qaeda did 
publicly claim responsibility for its occurrence: 

♦ August 18, 2003: Al-Hayat, an Egyptian media 
outlet, published excerpts from a communique 
attributed to al-Qaeda. Al Hayat claimed to have 
obtained the communique from the website of 
the International Islamic Media Center. The 
content of the communique asserts that the "bri- 
gades of Abu Fahes Al Masri had hit two main 
power plants supplying the East of the U.S., as 
well as major industrial cities in the U.S. and 
Canada, ‘its ally in the war against Islam (New 
York and Toronto) and their neighbors.”' Fur- 
thermore, the operation “was carried out on the 
orders of Osama bin Laden to hit the pillars of 
the U.S. economy," as “a realization of bin 
Laden’s promise to offer thejraqi people a pres- 
ent.” The communique does not specify the way 
in which the alleged sabotage was carried out, 
but it does elaborate on the alleged damage to 
the U.S. economy in the areas of finance, trans- 
portation, energy, and telecommunications. 

Additional claims and commentary regarding the 
power outage appeared in various Middle Eastern 
media outlets: 

♦ August 26, 2003: A conservative Iranian daily 
newspaper published a commentary regarding 
the potential of computer technology as a tool 
for terrorists against infrastructures dependent 
on computer networks-most notably, water, 
electric, public transportation, trade organiza- 
tions, and “supranational companies” in the 
United States. 

♦ September 4, 2003: An Islamist participant in a 
Jihadist chat room forum claimed that sleeper 


cells associated with al-Qaeda used the power 
outage as a cover to infiltrate the United States 
from Canada. 

These claims above, as known, are not consistent 
with the SWG’s findings to date. They are also not 
consistent with recent congressional testimony by 
the U.S. Federal Bureau of Investigation (FBI). 
Larry A. Mefford, Executive Assistant Director in 
charge of the FBI’s Counterterrorism and 
Counterintelligence programs, testified to the U.S. 
Congress on September 4, 2003, that, “To date, we 
have not discovered any evidence indicating that 
the outage was a result of activity by international 
or domestic terrorists or other criminal activity.” 
He also testified that, “The FBI has received no 
specific, credible threats to electronic power grids 
in the United States in the recent past and the 
claim of the Abu Hafs al-Masri Brigade to have 
caused the blackout appears to be no more than 
wishful thinking. We have no information con- 
firming the actual existence of this group.” Mr. 
Mefford’s Statement for the Record is available at 
web site http://www.fbi.gov/congress/congress03/ 
meffordD90403.htm. 

Current assessments suggest that there are terror- 
ists and other malicious actors who have the capa- 
bility to conduct a malicious cyber attack with 
potential to disrupt the energy infrastructure. 
Although such an attack cannot be ruled out 
entirely, an examination of available information 
and intelligence does not support any claims of a 
deliberate attack against the energy infrastructure 
on, or leading up to, August 14, 2003. The few 
instances of physical damage that occurred on 
power delivery lines were the result of natural acts 
and not of sabotage. No intelligence reports before, 
during, or after the power outage indicate any spe- 
cific terrorist plans or operations against the 
energy infrastructure. No incident reports detail 
suspicious activity near the power generation 
plants or delivery lines in question. 

Physical Analysis 

The Physical Analysis sub-team is led by the U.S. 
Secret Service and the RCMP. These organizations 
have particular expertise in physical security 
assessments in the energy sector. The sub-team is 
focusing on issues related to how the cyber-related 
facilities of the energy sector companies are 
secured, including the physical integrity of data 
centers and control rooms, along with security 
procedures and policies used to limit access to 
sensitive areas. Focusing on the facilities identi- 
fied as having a causal relationship to the outage, 
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the sub-team is seeking to determine whether the 
physical integrity of the cyber facilities was 
breached, either externally or by an insider, before 
or during the outage; and if so, whether such a 
breach caused or contributed to the power outage. 
Although the sub-team has analyzed information 
provided to both the EWG and the Nuclear 
Working Group (NWG), the Physical Analysis 
sub-team is also reviewing information resulting 
from recent face-to-face meetings with energy sec- 
tor personnel and site visits to energy sector facili- 
ties, to determine the physical integrity of the 
cyber infrastructure. 

The sub-team has compiled a list of questions cov- 
ering location, accessibility, cameras, alarms, 
locks, and fire protection and water systems as 
they apply to computer server rooms. Based on 
discussions of these questions during its inter- 
views, the sub-team is in the process of ascertain- 
ing whether the physical integrity of the cyber 
infrastructure was breached. Additionally, the 
sub-team is examining access and control mea- 
sures used to allow entry into command and con- 
trol facilities and the integrity of remote facilities. 

The sub-team is also concentrating on mecha- 
nisms used by the companies to report unusual 
incidents within server rooms, command and con- 
trol rooms, and remote facilities. The sub-team is 
also addressing the possibility of an insider attack 
on the cyber infrastructure. 

Policies and Procedures 

The Policies and Procedures sub-team is led by 
DHS and OCIPEP, which have personnel with 
strong backgrounds in the fields of electric deliv- 
ery operations, automated control systems 
(including SCADA and EMS), and information 
security. The sub-team is focused on examining 
the overall policies and procedures that may or 
may not have been in place during the events lead- 
ing up to and during the August 14 power outage. 
The team is examining policies that are centrally 
related to the cyber systems of the companies 
identified in the early stages of the power outage. 
Of specific interest are policies and procedures 
regarding the upgrade and maintenance (to 
include system patching) of the command and 
control (C2) systems, including SCADA and EMS. 
Also of interest are the procedures for contingency 
operations and restoration of systems in the event 
of a computer system failure or a cyber event, such 
as an active hack or the discovery of malicious 
code. The group is conducting further interviews 


and is continuing its analysis to build solid 
conclusions about the policies and procedures 
relating to the outage. 

Supporting Infrastructure 

The Supporting Infrastructure sub-team is led by a 
DHS expert with experience assessing supporting 
infrastructure elements such as water cooling for 
computer systems, backup power systems, heat- 
ing, ventilation and air conditioning (HVAC), and 
supporting telecommunications networks. 
OCIPEP is the Canadian co-lead for this effort. The 
sub-team is analyzing the integrity of the support- 
ing infrastructure and its role, if any, in the August 
14 power outage, and whether the supporting 
infrastructure was performing at a satisfactory 
level before and during the outage. In addition, the 
team is contacting vendors to determine whether 
there were maintenance issues that may have 
affected operations during or before the outage. 

The sub-team is focusing specifically on the fol- 
lowing key issues in visits to each of the desig- 
nated electrical entities; 

♦ Carrier/provider/vendor for the supporting 
infrastructure services and/or systems at select 
company facilities 

♦ Loss of service before and/or after the power 
outage 

♦ Conduct of maintenance activities before and/or 
after the power outage 

♦ Conduct of installation activities before and/or 
after the power outage 

♦ Conduct of testingactivities before and/or after 
the power outage 

♦ Conduct of exercises before and/or after the 
power outage 

♦ Existence of a monitoring process (log, check- 
list, etc.) to document the status of supporting 
infrastructure services. 

Root Cause Analysis 

The SWG Root Cause Liaison sub-team (SWG/RC) 
has been following the work of the ESWG to iden- 
tify potential root causes of the power outage. As 
these root cause elements are identified, the 
sub-team will assess with the ESWG any potential 
linkages to physical and/or cyber malfeasance. 

The root cause analysis work of the ESWG is still 
in progress; however, the initial analysis has 
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found no causal link between the power outage 
and malicious activity, whether physical or cyber 
initiated. Root cause analysis for an event like the 
August 14 power outage involves a detailed pro- 
cess to develop a hierarchy of actions and events 
that suggest causal factors. The process includes: 
development of a detailed timeline of the events, 
examination of actions related to the events, and 
an assessment of factors that initiated or exacer- 
bated the events. An assessment of the impact of 
physical security as a contributor to the power 
outage is conditional upon discovery of informa- 
tion suggesting that a malicious physical act initi- 
ated or exacerbated the power outage. There are 
no such indications thus far, and no further 
assessment by the SWG in this area is indicated. 

Cyber Timeline 

The following sequence of events was derived 
from discussions with representatives of 
FirstEnergy and the Midwest Independent Trans- 
mission System Operator (MISO). All times are 
approximate and will need to be confirmed by an 
analysis of company log data. 

♦ The first significant cyber-related event of 
August 14, 2003, occurred at 12:40 EDT at the 
MISO. At this time, a MISO EMS engineer pur- 
posely disabled the automatic periodic trigger 
on the State Estimator (SE) application, which 
allows MISO to determine the real-time state of 
the power system for its region. Disabling of the 
automatic periodic trigger, a program feature 
that causes the SE to run_automatically every 5 
minutes, is a necessary operating procedure 
when resolving a mismatched solution pro- 
duced by the SE. The EMS engineer determined 
that the mismatch in the SE solution was due to 
the SE model depicting Cinergy's Blooming- 
ton-Denois Creek 230-kV line as being in ser- 
vice, when it had actually been out of service 
since 12:12 EDT. 

♦ At 13:00 EDT, after making the appropriate 
changes to the SE model and manually trigger- 
ing the SE, the MISO EMS engineer achieved 
two valid solutions. 

♦ At 13:30 EDT, the MISO EMS engineer went to 
lunch. He forgot to re-engage the automatic 
periodic trigger. 

♦ At 14:14 EDT, FirstEnergy’s “Alarm and Event 
Processing Routine" (AEPR)-a key software pro- 
gram that gives operators visual and audible 
indications of events occurring on their portion 


of the grid-began to malfunction. FirstEnergy 
system operators were unaware that the soft- 
ware was not functioning properly. This soft- 
ware did not become functional again until 
much later that evening. 

♦ At 14:40 EDT, an Ops engineer discovered that 
the SE was not solving. He went to notify an 
EMS engineer. 

♦ At 14:41 EDT, FirstEnergy's server running the 
AEPR software failed to the backup server. Con- 
trol room staff remained unaware that the AEPR 
software was not functioning properly. 

♦ At 14:44 EDT, an MISO EMS engineer, after 
being alerted by the Ops engineer, reactivated 
the automatic periodic trigger and, for speed, 
manually triggered the program. The SE pro- 
gram again showed a mismatch. 

♦ At 14:54 EDT, FirstEnergy’s backup server 
failed. AEPR continued to malfunction. The 
Area Control Error (ACE) calculations and Strip 
Charting routines malfunctioned, and the dis- 
patcher user interface slowed significantly. 

♦ At 15:00 EDT, FirstEnergy used its emergency 
backup system to control the system and make 
ACE calculations. ACE calculations and control 
systems continued to run on the emergency 
backup system until roughly 15:08 EDT, when 
the primary server was restored. 

♦ At 15:05 EDT, FirstEnergy’s Harding-Chamber- 
lin 345-kV line tripped and locked out. FE sys- 
tem operators did not receive notification from 
the AEPR software, which continued to mal- 
function, unbeknownst to the FE system 
operators. 

♦ At 15:08 EDT, using data obtained at roughly 
15:04 EDT (it takes about 5 minutes for the SE to 
provide a result), the MISO EMS engineer con- 
cluded that the SE mismatched due to a line 
outage. His experience allowed him to isolate 
the outage to the Stuart-Atlanta 345-kV line 
(which tripped about an hour earlier, at 14:02 
EDT). He took the Stuart-Atlanta line out of ser- 
vice in the SE model and got a valid solution. 

♦ Also at 15:08 EDT, the FirstEnergy primary 
server was restored. ACE calculations and con- 
trol systems were now running on the primary 
server. AEPR continued to malfunction, unbe- 
knownst to the FirstEnergy system operators. 

♦ At 15:09 EDT, the MISO EMS engineer went to 
the control room to tell the operators that he 
thought the Stuart-Atlanta line was out of ser- 
vice. Control room operators referred to their 
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“Outage Scheduler” and informed the EMS 
engineer that their data showed the Stu- 
art-Atlanta line was “up” and that the EMS engi- 
neer should depict the line as in service in the 
SE model. At 15:17 EDT, the EMS engineer ran 
the SE with the Stuart-Atlanta line “live.” The 
model again mismatched. 

♦ At 15:29 EDT, the MISO EMS Engineer asked 
MISO operators to call the PJM Interconnect to 
determine the status of the Stuart-Atlanta line. 
MISO was informed that the Stuart-Atlanta line 
had tripped at 14:02 EDT. The EMS engineer 
adjusted the model, which by that time had 
been updated with the 15:05 EDT Har- 
ding-Chamberlin 345-kV line trip, and came up 
with a valid solution. 

♦ At 15:32 EDT, FirstEnergy's Hanna-Juniper 
345-kV line tripped and locked out. The AEPR 
continued to malfunction. 

♦ At 15:41 EDT, the lights flickered at 
FirstEnergy’s control facility, because the facil- 
ity had lost grid power and switched over to its 
emergency power supply. 

♦ At 15:42 EDT, a FirstEnergy dispatcher realized 
that the AEPR was not working and informed 
technical support staff of the problem. 

Findings to Date 

The SWG has developed the following findings 
via analysis of collected data and discussions with 
energy companies and entities identified by the 
E SWG as pertinent to the SWG’s analysis. SWG 
analysis to date provides no evidence that mali- 
cious actors-either individuals or organiza- 
tions-are responsible for, or contributed to, the 
power outage of August 14, 2003. The SWG con- 
tinues to coordinate closely with the other Task 
Force Working Groups and members of the U.S. 
and Canadian law enforcement and DHS/OCIPEP 
communities to collect and analyze data to test 
this preliminary finding. 

No intelligence reports before, during, or after the 
power outage indicated any specific terrorist plans 
or operations against the energy infrastructure. 
There was, however, threat information of a gen- 
eral nature related to the sector, which was pro- 
vided to the North American energy industry by 


U.S. and Canadian government agencies in late 
July 2003. This information indicated that 
al-Qaeda might attempt to carry out a physical 
attack against oil production facilities, power 
plants, or nuclear plants on the U.S. East Coast 
during the summer of 2003. The type of physical 
attack described in the intelligence that prompted 
the threat information was not consistent with the 
events of the power outage. 

Although there were a number of worms and 
viruses impacting the Internet and Internet- 
connected systems and networks in North Amer- 
ica before and during the outage, the SWG's pre- 
liminary analysis provides no indication that 
worm/virus activity had a significant effect on the 
power generation and delivery systems. Further 
SWG analysis will test this finding. 

SWG analysis to date suggests that failure of a soft- 
ware program-not linked to malicious activ- 
ity-may have contributed significantly to the 
power outage of August 14, 2003. Specifically, key 
personnel may not have been aware of the need to 
take preventive measures at critical times, because 
an alarm system was malfunctioning. The SWG 
continues to work closely with the operators of the 
affected system to determine the nature and scope 
of the failure, and whether similar software fail- 
ures could create future system vulnerabilities. 
The SWG is in the process of engaging system ven- 
dors and operators to determine whether any tech- 
nical or process-related modifications should be 
implemented to improve system performance in 
the future. 

The existence of both internal and external links 
— from SCADA systems to other systems introduced 
vulnerabilities. At this time, however, preliminary 
analysis of information derived from interviews 
with operators provides no evidence indicating 
exploitation of these vulnerabilities before or dur- 
ing the outage. Future SWG work will provide 
greater insight into this issue. 

Analysis of information derived from interviews 
with operators suggests that, in some cases, visi- 
bility into the operations of surrounding areas was 
lacking. Some companies appear to have had only 
a limited understanding of the status of the elec- 
tric systems outside their immediate control. This 
may have been, in part, the result of a failure to use 
modern dynamic mapping and data sharing sys- 
tems. Future SWG work will clarify this issue. 
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Appendix A 


Description of Outage Investigation and 
Plan for Development of Recommendations 


On August 14, 2003, the northeastern U.S. and 
Ontario, Canada, suffered one of the largest power 
blackouts in the history of North America. The 
area affected extended from New York, Massachu- 
setts, and New Jersey west to Michigan, and from 
Ohio north to Ontario. 

This appendix outlines the process used to inves- 
tigate why the blackout occurred and was not con- 
tained, and explains how recommendations will 
be developed to prevent and minimize the scope 
of future outages. The essential first step in the 
process was the creation of a joint U.S.-Canada 
Power System Outage Task Force to provide over- 
sight for the investigation and the development of 
recommendations. 

Task Force Composition and 
Responsibilities 

President George W. Bush and Prime Minister 
Jean Chrdtien created the joint Task Force to iden- 
tify the causes of the August 14, 2003 power out- 
age and to develop recommendations to prevent 
and contain future outages. The co-chairs of the 
Task Force are U.S. Secretary of Energy Spencer 
Abraham and Minister of Natural Resources Can- 
ada Herb Dhaliwal. Other U.S. members are Nils J. 
Diaz, Chairman of the Nuclear Regulatory Com- 
mission, Tom Ridge, Secretary of Homeland Secu- 
rity, and Pat Wood, Chairman of the Federal 
Energy Regulatory Commission. The other Cana- 
dian members are Deputy Prime Minister John 
Manley, Linda J. Keen, President and CEO of the 
Canadian Nuclear Safety Commission, and Ken- 
neth Vollman, Chairman of the National Energy 
Board. The coordinators for the Task Force are 
Jimmy Glotfelty on behalf of the U.S. Department 
of Energy and Dr. Nawal Kamel on behalf of Natu- 
ral Resources Canada. 

U.S. Energy Secretary Spencer Abraham and Min- 
ister of Natural Resources Canada Herb Dhaliwal 
met in Detroit, Michigan on August 20, and agreed 
on an outline for the Task Force’s activities. The 
outline directed the Task Force to divide its efforts 
into two phases. The first phase was to focus on 
what caused the outage and why it was not con- 
tained, and the second was to focus on the 


development of recommendations to prevent and 
minimize future power outages. On August 27, 
Secretary Abraham and Minister Dhaliwal 
announced the formation of three Working 
Groups to support the work of the Task Force. The 
three Working Groups address electric system 
issues, security matters, and questions related to 
the performance of nuclear power plants over the 
course of the outage. The members of the Working 
Groups are officials from relevant federal depart- 
ments and agencies, technical experts, and senior 
representatives from the affected states and the 
Province of Ontario. 

U.S.-Canada-NERC Investigation Team 

Under the oversight of the Task Force, a team of 
electric system experts was established to investi- 
gate the causes of the outage. This team was com- 
prised of individuals from several U.S. federal 
agencies, the U.S. Department of Energy’s national 
laboratories, Canadian electric industry, Canada's 
National Energy Board, staff from the North Amer- 
ican Electric Reliability Council (NERC), and the 
U.S. electricity industry. The overall investigative 
team was divided into several analytic groups 
with specific responsibilities, including data man- 
agement, determining the sequence of outage 
events, system modeling, evaluation of operating 
tools and communications, transmission system 
performance, generator performance, vegetation 
and right-of-way management, transmission and 
reliability investments, and root cause analysis. 
The root cause analysis is best understood as an 
analytic framework as opposed to a stand-alone 
analytic effort. Its function was to enable the ana- 
lysts to draw upon and organize information from 
all of the other analyses, and by means of a rigor- 
ously logical and systematic procedure, assess 
alternative hypotheses and identify the root 
causes of the outage. 

Separate teams were established to address issues 
related to the performance of nuclear power plants 
affected by the outage, and physical and cyber 
security issues related to the bulk power 
infrastructure. 
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Function of the Working Groups 

The U.S. and Canadian co-chairs of each of the 
three Working Groups (i.e., an Electric System 
Working Group, a Nuclear Working Group, and a 
Security Working Group) designed various work 
products to be prepared by the investigative 
teams. Drafts of these work products were 
reviewed and commented upon by the relevant 
Working Groups. These work products were then 
synthesized into a single Interim Report reflecting 
the conclusions of the three investigative teams 
and the Working Groups. Determination of when 
the Interim Report was complete and appropriate 
for release to the public was the responsibility of 
the joint Task Force. 

Confidentiality of Data and Information 

Given the seriousness of the blackout and the 
importance of averting or minimizing future 
blackouts, it was essential that the Task Force’s 
teams have access to pertinent records and data 
from the regional independent system operators 
(ISOs) and electric companies affected by the 
blackout, and for the investigative team to be able 
to interview appropriate individuals to learn what 
they saw and knew at key points in the evolution 
of the outage, what actions they took, and with 
what purpose. In recognition of the sensitivity of 
this information, Working Group members and 
members of the teams signed agreements affirm- 
ing that they would maintain the confidentiality of 
data and information provided to them, and 
refrain from independent or premature statements 
to the media or the public about the activities, 
findings, or conclusions of the individual Working 
Groups or the Task Force as a whole. 

Relevant U.S. and Canadian Legal 
Framework 

United States 

The Secretary of Energy directed the Department 
of Energy (DOE) to gather information and con- 
duct an investigation to examine the cause or 
causes of the August 14, 2003 blackout. In initiat- 
ing this effort, the Secretary exercised his author- 
ity, including section 11 of the Energy Supply and 
Environmental Coordination Act of 1974, and sec- 
tion 13 of the Federal Energy Administration Act 
of 1974, to gather energy-related information and 
conduct investigations. This authority gives him 
and the DOE the ability to collect such energy 
information as he deems necessary to assist in the 


formulation of energy policy, to conduct investi- 
gations at reasonable times and in a reasonable 
manner, and to conduct physical inspections at 
energy facilities and business premises. In addi- 
tion, DOE can inventory and sample any stock of 
fuels or energy sources therein, inspect and copy 
records, reports, and documents from which 
energy information has been or is being compiled 
and to question such persons as it deems neces- 
sary. DOE worked closely with the Canadian 
Department of Natural Resources and NERC on 
the investigation. 

Canada 

Minister Dhaliwal, as the Minister responsible for 
Natural Resources Canada, was appointed by 
Prime Minister Chretien as the Canadian Co-Chair 
of the Task Force. Minister Dhaliwal works closely 
with his American Co-Chair, Secretary of Energy 
Abraham, as well as NERC and his provincial 
counterparts in carrying out his responsibilities. 
The Task Force will report to the Prime Minister 
and the US President upon the completion of its 
mandate. 

Under Canadian law, the Task Force is character- 
ized as a non-statutory, advisory body that does 
not have independent legal personality. The Task 
Force does not have any power to compel evi- 
dence or witnesses, nor is it able to conduct 
searches or seizures. In Canada, the Task Force 
will rely on voluntary disclosure for obtaining 
information pertinent to its work. 

Investigative Process 

Collection of Data and Information from ISOs , 
Utilities, States, and the Province of Ontario 

On Tuesday, August 19, 2003, investigators affili- 
ated with the U.S. Department of Energy (USDOE) 
began interviewing control room operators and 
other key officials at the ISOs and the companies 
most directly involved with the initial stages of the 
outage. In addition to the information gained in 
the interviews, the interviewers sought informa- 
tion and data about control room operations and 
practices, the organization's system status and 
conditions on August 14, the organization’s oper- 
ating procedures and guidelines, load limits on its 
system, emergency planning and procedures, sys- 
tem security analysis tools and procedures, and 
practices for voltage and frequency monitoring. 
Similar interviews were held later with staff at 
Ontario’s Independent Electricity Market Opera- 
tor (IMO) and Hydro One in Canada. 
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On August 22 and 26, NERC directed the reliabil- 
ity coordinators at the ISOs to obtain a wide range 
of data and information from the control area coor- 
dinators under their oversight. The data requested 
included System Control and Data Acquisition 
(SCADA) logs. Energy Management System (EMS) 
logs, alarm logs, data from local digital fault 
recorders, data on transmission line and generator 
“trips” {i.e., automatic disconnection to prevent 
physical damage to equipment), state estimator 
data, operator logs and transcripts, and informa- 
tion related to the operation of capacitors, phase 
shifting transformers, load shedding, static var 
compensators, special protection schemes or sta- 
bility controls, and high-voltage direct current 
(HVDC) facilities. NERC issued another data 
request to FirstEnergy on September 15 for copies 
of studies since 1990 addressing voltage support, 
reactive power supply, static capacitor applica- 
tions, voltage requirements, import or transfer 
capabilities {in relation to reactive capability or 
voltage levels), and system impacts associated 
with unavailability of the Davis-Besse plant. All 
parties were instructed that data and information 
provided to either DOE or NERC did not have to be 
submitted a second time to the other entity — all 
material provided would go into a common data 
base. 

The investigative team held three technical con- 
ferences (August 22, September 8-9, and October 
1-3) with the ISOs and key utilities aimed at clari- 
fying the data received, filling remaining gaps in 
the data, and developing a shared understanding 
of the data’s implications. The team also requested 
information from the public utility commissions 
in the affected states and Ontario on transmission 
right-of-way maintenance, transmission planning, 
and the scope of any state-led investigations con- 
cerning the August 14 blackout. The team also 
commissioned a study by a firm specializing in 
utility vegetation management to identify “best 
practices” concerning such management in right 
of way areas and to use those practices in gauging 
the performance of companies who had lines go 
out of service on August 14 due to tree contact. 

Data “ Warehouse ” 

The data collected by the investigative team 
became voluminous, so an electronic repository 
capable of storing thousands of transcripts, 
graphs, generator and transmission data and 
reports was constructed in Princeton, NJ at the 
NERC headquarters. At present the data base is 
over 20 Gigabytes of information. That data 


consists of over 10,000 different files some of 
which contain multiple files. The objective was to 
establish a set of validated databases that the sev- 
eral analytic teams could access independently on 
an as-needed basis. 

The following are the information sources for the 
Electric System Investigation: 

♦ Interviews conducted by members of the 
U.S.-Canada Electric Power System Outage 
Investigation Team with personnel at all of the 
utilities, control areas and reliability coordina- 
tors in the weeks following the blackout. 

♦ Three fact-gathering meetings conducted by the 
Investigation Team with personnel from the 
above organizations on August 22, September 8 
and 9, and October 1 to 3, 2003. 

♦ Materials provided by the above organizations 
in response to one or more data requests from 
the Investigation Team. 

♦ Extensive review of all taped phone transcripts 
between involved operations centers. 

♦ Additional interviews and field visits with oper- 
ating personnel on specific issues in October, 
2003. 

♦ Field visits to examine transmission lines and 
vegetation at short-circuit locations. 

. ♦ Materials provided by utilities and state regula- 
tors in response to data requests on vegetation 
management issues. 

♦ Detailed examination of thousands of individ- 
ual relay trips for transmission and generation 
events. 

♦ Computer simulation and modeling conducted 
by groups of experts from utilities, reliability 
coordinators, reliability councils, and the U.S. 
and Canadian governments. 

Sequence of Events 

Establishing a precise and accurate sequence of 
outage-related events was a critical building block 
for the other parts of the investigation. One of the 
key problems in developing this sequence was 
that although much of the data pertinent to an 
event was time-stamped, there was some variance 
from source to source in how the time -stamping 
was done, and not all of the time-stamps were syn- 
chronized to the National Institute of Standards 
and Technology (NIST) standard clock in Boulder, 
CO. Validating the timing of specific events 
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became a large, important, and sometimes diffi- 
cult task. This work was also critical to the issu- 
ance by the Task Force on September 12 of a 
“timeline” for the outage. The timeline briefly 
described the principal events, in sequence, lead- 
ing up to the initiation of the outage’s cascade 
phase, and then in the cascade itself. The timeline 
was not intended, however, to address the causal 
relationships among the events described, or to 
assign fault or responsibility for the blackout. All 
times in the chronology are in Eastern Daylight 
Time. 

System Modeling and Simulation Analysis 

The system modeling and simulation team repli- 
cated system conditions on August 14 and the 
events leading up to the blackout. While the 
sequence of events provides a precise description 
of discrete events, it does not describe the overall 
state of the electric system and how close it was to 
various steady-state, voltage stability, and power 
angle stability limits. An accurate computer 
model of the system, benchmarked to actual con- 
ditions at selected critical times on August 14, 
allowed analysts to conduct a series of sensitivity 
studies to determine if the system was stable and 
within limits at each point in time leading up to 
the cascade. The analysis also confirmed when the 
system became unstable, and allowed analysts to 
test whether measures such as load-shedding 
would have prevented the cascade. 

This team consisted of a number of NERC staff and 
persons with expertise in areas necessary to read 
and interpret all of the data logs, digital fault 
recorder information, sequence of events record- 
ers information, etc. The team consisted of about 
36 people involved at various different times with 
additional experts from the affected areas to 
understand the data. 

Assessment of Operations Tools, SCADA/EMS, 
Communications, and Operations Planning 

The Operations Tools, SCADA/EMS, Communica- 
tions, and Operations Planning Team assessed the 
observability of the electric system to operators 
and reliability coordinators, and the availability 
and effectiveness of operational (real-time and 
day-ahead) reliability assessment tools, including 
redundancy of views and the ability to observe the 
“big picture” regarding bulk electric system condi- 
tions. The team investigated operating practices 
and effectiveness of operating entities and reliabil- 
ity coordinators in the affected area. This team 
investigated all aspects of the blackout related to 


operator and reliability coordinator knowledge of 
system conditions, action or inactions, and 
communications . 

Frequency! ACE Analysis 

The Frequency/ACE Team analyzed potential fre- 
quency anomalies that may have occurred on 
August 14, as compared to typical interconnection 
operations. The team also determined whether 
there were any unusual issues with control perfor- 
mance and frequency and any effects they may 
have had related to the cascading failure, and 
whether frequency related anomalies were con- 
tributing factors or symptoms of other problems 
leading to the cascade. 

Assessment of Transmission System 
Performance, Protection, Control, 

Maintenance, and Damage 
This team investigated the causes of all transmis- 
sion facility automatic operations (trips and 
reclosings) leading up to and through to the end of 
the cascade on all facilities greater than 100 kV. 
Included in the review were relay protection and 
remedial action schemes and identification of the 
cause of each operation and any misoperations 
that may have occurred. The team also assessed 
transmission facility maintenance practices in the 
affected area as compared to good utility practice 
and identified any transmission equipment that 
was damaged in any way as a result of the cascad- 
ing outage. The team reported patterns and con- 
clusions regarding what caused transmission 
facilities to trip; why did the cascade extend as far 
as it did and not further into other systems; any 
misoperations and the effect those misoperations 
had on the outage; and any transmission equip- 
ment damage. Also the team reported on the trans- 
mission facility maintenance practices of entities 
in the affected area compared to good utility 
practice. 

Assessment of Generator Performance, 
Protection, Controls, Maintenance, and 
Damage 

This team investigated the cause of generator trips 
for all generators with a 10 MW or greater name- 
plate rating leading to and through the end of the 
cascade. The review included the cause for the 
generator trips, relay targets, unit power runbacks, 
and voltage/reactive power excursions. The team 
reported any generator equipment that was dam- 
aged as a result of the cascading outage. The team 
reported on patterns and conclusions regarding 
what caused generation facilities to trip. The team 
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identified any unexpected performance anomalies 
or unexplained events. The team assessed genera- 
tor maintenance practices in the affected area as 
compared to good utility practice. The team 
analyzed the coordination of generator under- 
frequency settings with transmission settings, 
such as under-frequency load shedding. The team 
gathered and analyzed data on affected nuclear 
units and worked with the Nuclear Regulatory 
Commission to address U.S. nuclear unit issues. 

Assessment of Right of Way (ROW) 
Maintenance 

The Vegetation/ROW Team investigated the prac- 
tices of transmission facilities owners in the 
affected areas for vegetation management and 
ROW maintenance. These practices were com- 
pared to accepted utility practices in general 
across the Eastern Interconnection. Also, the team 
investigated historical patterns in the area related 
to outages caused by contact with vegetation. 

Root Cause Analysis 

The investigation team used an analytic technique 
called root cause analysis to help guide the overall 
investigation process by providing a systematic 


approach to evaluating root causes and contribut- 
ing factors leading to the start of the cascade on 
August 14. The root cause analysis team worked 
closely with the technical investigation teams pro- 
viding feedback and queries on additional infor- 
mation. Also, drawing on other data sources as 
needed, the root cause analysis verified facts 
regarding conditions and actions (or inactions) 
that contributed to the blackout. 

Oversight and Coordination 

The Task Force’s U.S. and Canadian coordinators 
held frequent conference calls to ensure that all 
components of the investigation were making 
timely progress. They briefed both Secretary Abra- 
ham and Minister Dhaliwal regularly and pro- 
vided weekly summaries from all components on 
the progress of the investigation. The leadership of 
the electric system investigation team held daily 
conference calls to address analystical and pro- 
cess issues through the investigation. The three 
Working Groups held weekly conference calls to 
enable the investigation team to update the 
Working Group members on the state of the over- 
all analysis. 
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Appendix B 

List of Electricity Acronyms 


BPA Bonneville Power Administration 

CNSC Canadian Nuclear Safety Commission 

DOE Department of Energy (U.S.) 

ECAR East Central Area Reliability Coordination Agreement 

ERCOT Electric Reliability Council of Texas 

FERC Federal Energy Regulatory Commission (U.S.) 

FRCC Florida Reliability Coordinating Council 

GW, GWh Gigawatt, Gigawatt-hour 

kV, kVAr Kilovolt, Kilovolt-amperes-reactive 

kW, kWh Kilowatt, Kilowatt-hour 

MAAC Mid-Atlantic Area Council 

MAIN Mid-America Interconnected Network 

MAPP Mid-Continent Area Power Pool 

MV A, MVAr Megavolt-amperes, Megavolt-amperes-reactive 

MW, MWh Megawatt, Megawatt-hour 

NERC North American Electric Reliability Council 

NPCC Northeast Power Coordination Council 

NRC Nuclear Regulatory Commission (U.S.) 

NRCan Natural Resources Canada 

OTD Office of Transmission and Distribution (U.S. DOE) 

PUC Public Utility Commission (state) 

RTO Regional Transmission Organization 

SERC Southeast Electric Reliability Council 

SPP Southwest Power Pool 


TVA Tennessee Valley Authority (U.S.) 
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Appendix C 

Electricity Glossary 


AC: Alternating current; current that changes peri- 
odically (sinusoidally) with time. 

ACE: Area Control Error in MW. A negative value 
indicates a condition of under-generation relative 
to system load and Imports, and a positive value 
denotes over-generation. 

Active Power: Also known as “real power.” The 
rate at which work is performed or that energy is 
transferred. Electric power is commonly measured 
in watts or kilowatts. The terms “active” or “real” 
power are often used in place of the term power 
alone to differentiate it from reactive power. The 
rate of producing, transferring, or using electrical 
energy, usually expressed in kilowatts (kW) or 
megawatts (MW). 

Adequacy: The ability of the electric system to 
supply the aggregate electrical demand and energy 
requirements of customers at all times, taking into 
account scheduled and reasonably expected 
unscheduled outages of system elements. 

AGC: Automatic Generation Control is a computa- 
tion based on measured frequency and computed 
economic dispatch. Generation equipment under 
AGC automatically respond to signals from an 
EMS computer in real time to adjust power output 
in response to a change in system frequency, 
tie-line loading, or to a prescribed relation 
between these quantities. Generator output is^ 
adjusted so as to maintain a target system fre- 
quency (usually 60 Hz) and any scheduled MW 
interchange with other areas. 

Apparent Power: The product of voltage and cur- 
rent phasors. It comprises both active and reactive 
power, usually expressed in kilovoltamperes 
(kVA) or megavoltamperes (MV A). 

Automatic Operating Systems: Special protection 
systems, or remedial action schemes, that require 
no intervention on the part of system operators. 

Blackstart Capability: The ability of a generating 
unit or station to go from a shutdown condition to 
an operating condition and start delivering power 
without assistance from the electric system. 

Bulk Electric System: A term commonly applied 
to the portion of an electric utility system that 


encompasses the electrical generation resources 
and bulk transmission system. 

Bulk Transmission: A functional or voltage classi- 
fication relating to the higher voltage portion of 
the transmission system, specifically, lines at or 
above a voltage level of 115 kV. 

Bus: Shortened from the word busbar, meaning a 
node in an electrical network where one or more 
elements are connected together. 

Capacitor Bank: A capacitor is an electrical device 
that provides reactive power to the system and is 
often used to compensate for reactive load and 
help support system voltage. A bank is a collection 
of one or more capacitors at a single location. 

Capacity: The rated continuous load-carrying 
ability, expressed in megawatts (MW) or 
megavolt-amperes (MV A) of generation, transmis- 
sion, or other electrical equipment. 

Cascading: The uncontrolled successive loss of 
system elements triggered by an incident at any 
location. Cascading results in widespread service 
interruption, which cannot be restrained from 
sequentially spreading beyond an area predeter- 
mined by appropriate studies. 

Circuit: A conductor or a system of conductors 
through which electric current flows. 

Circuit Breaker: A switching device connected to 
the end of a transmission line capable of opening 
or closing the circuit in response to a command, 
usually from a relay. 

Control Area: An electric power system or combi- 
nation of electric power systems to which a com- 
mon automatic control scheme is applied in order 
to: (1) match, at all times, the power output of the 
generators within the electric power system(s) and 
capacity and energy purchased from entities out- 
side the electric power system(s), with the load in 
the electric power system(s); (2) maintain, within 
the limits of Good Utility Practice, scheduled 
interchange with other Control Areas; (3) main- 
tain the frequency of the electric power system(s) 
within reasonable limits in accordance with 
Good Utility Practice; and (4) provide sufficient 
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generating capacity to maintain operating reserves 
in accordance with Good Utility Practice. 

Contingency: The unexpected failure or outage of 
a system component, such as a generator, trans- 
mission line, circuit breaker, switch, or other elec- 
trical element. A contingency also may include 
multiple components, which are related by situa- 
tions leading to simultaneous component outages. 

Control Area Operator: An individual or organi- 
zation responsible for controlling generation to 
maintain interchange schedule with other control 
areas and contributing to the frequency regulation 
of the interconnection. The control area is an elec- 
tric system that is bounded by interconnection 
metering and telemetry. 

Current (Electric): The rate of flow of electrons in 
an electrical conductor measured in Amperes. 

DC: Direct current; current that is steady and does 
not change with time. 

Dispatch Operator: Control of an integrated elec- 
tric system involving operations such as assign- 
ment of levels of output to specific generating 
stations and other sources of supply; control of 
transmission lines, substations, and equipment; 
operation of principal interties and switching; and 
scheduling of energy transactions. 

Distribution Network: The portion of an electric 
system that is dedicated to delivering electric 
energy to an end user, at or below 69 kV. The dis- 
tribution network consists primarily of low- 
voltage lines and transformers that “transport” 
electricity from the bulk power system to retail 
customers. — 

Disturbance: An unplanned event that produces 
an abnormal system condition. 

Electrical Energy: The generation or use of elec- 
tric power by a device over a period of time, 
expressed in kilowatthours (kWh), megawatt- 
hours (MWh), or gigawatthours (GWh). 

Electric Utility Corporation: Person, agency, 
authority, or other legal entity or instrumentality 
that owns or operates facilities for the generation, 
transmission, distribution, or sale of electric 
energy primarily for use by the public, and is 
defined as a utility under the statutes and rules by 
which it is regulated. An electric utility can be 
investor-owned, cooperatively owned, or govern- 
ment-owned (by a federal agency, crown corpora- 
tion, State, provincial government, municipal 
government, and public power district). 


Emergency: Any abnormal system condition that 
requires automatic or immediate manual action to 
prevent or limit loss of transmission facilities or 
generation supply that could adversely affect the 
reliability of the electric system. 

Emergency Voltage Limits: The operating voltage 
range on the interconnected systems that is 
acceptable for the time, sufficient for system 
adjustments to be made following a facility outage 
or system disturbance. 

EMS: An Energy Management System is a com- 
puter control system used by electric utility dis- 
patchers to monitor the real time performance of 
various elements of an electric system and to con- 
trol generation and transmission facilities. 

Fault: A fault usually means a short circuit, but 
more generally it refers to some abnormal system 
condition. Faults occur as random events, usually 
an act of nature. 

Federal Energy Regulatory Commission (FERC): 
Independent Federal agency within the U.S. 
Department of Energy that, among other responsi- 
bilities, regulates the transmission and wholesale 
sales of electricity in interstate commerce. 

Flashover: A plasma arc initiated by some event 
such as lightning. Its effect is a short circuit on the 
network. 

Flowgate: A single or group of transmission ele- 
ments intended to model MW flow impact relating 
to transmission limitations and transmission ser- 
vice usage. 

Forced Outage: The removal from service avail- 
ability of a generating unit, transmission line, or 
other facility for emergency reasons or a condition 
in which the equipment is unavailable due to 
unanticipated failure. 

Frequency: The number of complete alternations 
or cycles per second of an alternating current, 
measured in Hertz. The standard frequency in the 
United States is 60 Hz. In some other countries the 
standard is 50 Hz. 

Frequency Deviation or Error: A departure from 
scheduled frequency. The difference between 
actual system frequency and the scheduled sys- 
tem frequency. 

Frequency Regulation: The ability of a Control 
Area to assist the interconnected system in main- 
taining scheduled frequency. This assistance can 
include both turbine governor response and auto- 
matic generation control. 
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Frequency Swings: Constant changes in fre- 
quency from its nominal or steady-state value. 

Generation (Electricity): The process of produc- 
ing electrical energy from other forms of energy; 
also, the amount of electric energy produced, usu- 
ally expressed in kilowatt hours (kWh) or mega- 
watt hours (MWh), 

Generator: Generali, an electromechanical device 
used to convert mechanical power to electrical 
power. 

Grid: An electrical transmission and/or distribu- 
tion network. 

Grid Protection Scheme: Protection equipment 
for an electric power system, consisting of circuit 
breakers, certain equipment for measuring electri- 
cal quantities (e.g., current and voltage sensors) 
and devices called relays. Each relay is designed to 
protect the piece of equipment it has been 
assigned from damage. The basic philosophy in 
protection system design is that any equipment 
that is threatened with damage by a sustained 
fault is to be automatically taken out of service. 

Ground: A conducting connection between an 
electrical circuit or device and the earth. A ground 
may be intentional, as in the case of a safety 
ground, or accidental, which may result in high 
overcurrents. 

Imbalance: A condition where the generation and 
interchange schedules do not match demand. 

Impedance: The total effects of a circuit that 
oppose the flow of an alternating current consist- 
ing of inductance, capacitance, and resistance. It 
can be quantified in the units of ohms. 

Independent System Operator (ISO): An organi- 
zation responsible for the reliable operation of the 
power grid under its purview and for providing 
open transmission access to all market partici- 
pants on a nondiscriminatory basis. An ISO is 
usually not-for-profit and can advise other utilities 
within its territory on transmission expansion and 
maintenance but does not have the responsibility 
to carry out the functions. 

Interchange: Electric power or energy that flows 
across tie-lines from one entity to another, 
whether scheduled or inadvertent. 

Interconnected System: A system consisting of 
two or more individual electric systems that nor- 
mally operate in synchronism and have connect- 
ing tie lines. 


Interconnection: When capitalized, any one of the 
five major electric system networks in North 
America: Eastern, Western, ERCOT (Texas), Que- 
bec, and Alaska. When not capitalized, the facili- 
ties that connect two systems or Control Areas. 
Additionally, an interconnection refers to the 
facilities that connect a nonutility generator to a 
Control Area or system. 

Interface: The specific set of transmission ele- 
ments between two areas or between two areas 
comprising one or more electrical systems. 

Island: A portion of a power system or several 
power systems that is electrically separated from 
the interconnection due to the disconnection of 
transmission system elements. 

Kiiovar (kVAr): Unit of alternating current reac- 
tive power equal to 1,000 VArs. 

Kilovolt (kV): Unit of electrical potential equal to 
1,000 Volts. 

Kilovolt-Amperes (kVA): Unit of apparent power 
equal to 1,000 volt amperes. Here, apparent power 
is in contrast to real power. On ac systems the volt- 
age and current will not be in phase if reactive 
power is being transmitted. 

Kilowatthour (kWh): Unit of energy equaling one 
thousand watthours, or one kilowatt used over 
one hour. This is the normal quantity used for 
metering and billing electricity customers. The 
price for a kWh varies from approximately 4 cents 
to 15 cents. At a 100% conversion efficiency, one 
kWh is equivalent to about 4 fluid ounces of gaso- 
line, 3/16 pound of liquid petroleum, 3 cubic feet 
of natural gas, or 1/4 pound of coal. 

Line Trip: Refers to the automatic opening of the 
conducting path provided by a transmission line 
by the circuit breakers. These openings or “trips" 
are designed to protect the transmission line dur- 
ing faulted conditions. 

Load (Electric): The amount of electric power 
delivered or required at any specific point or 
points on a system. The requirement originates at 
the energy-consuming equipment of the consum- 
ers. Load should not be confused with demand, 
which is the measure of power that a load receives 
or requires. See "Demand.” 

Load Shedding: The process of deliberately 
removing (either manually or automatically) pre- 
selected customer demand from a power system in 
response to an abnormal condition, to maintain 
the integrity of the system and minimize overall 
customer outages. 
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Lockout: A state of a transmission line following 
breaker operations where the condition detected 
fay the protective relaying was not eliminated by 
temporarily opening and reclosing the line, possi- 
bly multiple times. In this state, the circuit break- 
ers cannot generally be reclosed without resetting 
a lockout device. 

Market Participant: An entity participating in the 
energy marketplace by buying/selling transmis- 
sion rights, energy, or ancillary services into, out 
of, or through an ISO-controlled grid. 

Megawatthour (MWh): One million watthours. 

NERC Interregional Security Network (ISN): A 
communications network used to exchange elec- 
tric system operating parameters in near real time 
among those responsible for reliable operations of 
the electric system. The ISN provides timely and 
accurate data and information exchange among 
reliability coordinators and other system opera- 
tors. The ISN, which operates over the frame relay 
NERCnet system, is a private Intranet that is capa- 
ble of handling additional applications between 
participants. 

Normal (Precontingency) Operating Procedures: 
Operating procedures that are normally invoked 
by the system operator to alleviate potential facil- 
ity overloads or other potential system problems 
in anticipation of a contingency. 

Normal Voltage Limits: The operating voltage 
range on the interconnected systems that is 
acceptable on a sustained basis. 

North American Electric Reliability Council 
(NERC): A not-for-profit company formed by the 
electric utility industry in 1968 to promote the 
reliability of the electricity supply in North Amer- 
ica. NERC consists of nine Regional Reliability 
Councils and one Affiliate, whose members 
account for virtually all the electricity supplied in 
the United States, Canada, and a portion of Baja 
California Norte, Mexico. The members of these 
Councils are from all segments of the electricity 
supply industry: investor-owned, federal, rural 
electric cooperative, state/municipal, and provin- 
cial utilities, independent power producers, and 
power marketers. The NERC Regions are: East 
Central Area Reliability Coordination Agreement 
(ECAR); Electric Reliability Council of Texas 
(ERCOT); Mid-Atlantic Area Council (MAAC); 
Mid- America Interconnected Network (MAIN); 
Mid-Continent Area Power Pool (MAPP); North- 
east Power Coordinating Council (NPCC); 


Southeastern Electric Reliability Council (SERC); 
Southwest Power Pool (SPP); Western Systems 
Coordinating Council (WSCC); and Alaskan Sys- 
tems Coordination Council (ASCC, Affiliate). 

Operating Criteria: The fundamental principles 
of reliable interconnected systems operation, 
adopted by NERC. 

Operating Guides: Operating practices that a Con- 
trol Area or systems functioning as part of a Con- 
trol Area may wish to consider. The application of 
Guides is optional and may vary among Control 
Areas to accommodate local conditions and indi- 
vidual system requirements. 

Operating Policies: The doctrine developed for 
interconnected systems operation. This doctrine 
consists of Criteria, Standards, Requirements, 
Guides, and instructions, which apply to all Con- 
trol Areas. 

Operating Procedures: A set of policies, practices, 
or system adjustments that may be automatically 
or manually implemented by the system operator 
within a specified time frame to maintain the 
operational integrity of the interconnected electric 
systems. 

Operating Requirements: Obligations of a Control 
Area and systems functioning as part of a Control 
Area. 

Operating Standards: The obligations of a Control 
Area and systems functioning as part of a Control 
Area that are measurable. An Operating Standard 
may specify monitoring and surveys for 
compliance. 

Outage: The period during which a generating 
unit, transmission line, or other facility is out of 
service. 

Post-contingency Operating Procedures: Oper- 
ating procedures that may be invoked by the sys- 
tem operator to mitigate or alleviate system 
problems after a contingency has occurred. 

Protective Relay: A device designed to detect 
abnormal system conditions, such as electrical 
shorts on the electric system or within generating 
plants, and initiate the operation of circuit break- 
ers or other control equipment. 

Power/Phase Angle: The angular relationship 
between an ac (sinusoidal) voltage across a circuit 
element and the ac (sinusoidal) current through it. 
The real power that can flow is related to this 
angle. 
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Power: See “Active Power.” 

Reactive Power: The portion of electricity that 
establishes and sustains the electric and magnetic 
fields of alternating-current equipment. Reactive 
power must be supplied to most types of magnetic 
equipment, such as motors and transformers. It 
also must supply the reactive losses on transmis- 
sion facilities. Reactive power is provided by gen- 
erators, synchronous condensers, or electrostatic 
equipment such as capacitors and directly influ- 
ences electric system voltage. It is usually 
expressed in kilovars {kVAr} or megavars (MVAr). 
The mathematical product of voltage and current 
consumed by reactive loads. Examples of reactive 
loads include capacitors and inductors. These 
types of loads, when connected to an ac voltage 
source, will draw current, but because the current 
is 90 degrees out of phase with the applied voltage, 
they actually consume no real power in the ideal 
sense. 

Real Power: See "Active Power.” 

Regional Transmission Operator (RTO): An orga- 
nization that is independent from all generation 
and power marketing interests and has exclusive 
responsibility for electric transmission grid opera- 
tions, short-term electric reliability, and transmis- 
sion services within a multi-State region. To 
achieve those objectives, the RTO manages trans- 
mission facilities owned by different companies 
and encompassing one, large, contiguous geo- 
graphic area. 

Relay: A device that controls the opening and sub- 
sequent reclosing of circuit breakers. Relays take 
measurements from local current and voltage 
transformers, and from communication channels 
connected to the remote end of the lines. A relay 
output trip signal is sent to circuit breakers when 
needed. 

Relay Setting: The parameters that determine 
when a protective relay will initiate operation of 
circuit breakers or other control equipment. 
Reliability: The degree of performance of the ele- 
ments of the bulk electric system that results in 
electricity being delivered to customers within 
accepted standards and in the amount desired. 
Reliability may be measured by the frequency, 
duration, and magnitude of adverse effects on the 
electric supply. Electric system reliability can be 
addressed by considering two basic and func- 
tional aspects of the electric system Adequacy and 
Security. 

Reliability Coordinator: An individual or organi- 
zation responsible for the safe and reliable 


operation of the interconnected transmission sys- 
tem for their defined area, in accordance with 
NERC reliability standards, regional criteria, and 
subregional criteria and practices. 

Resistance: The characteristic of materials to 
restrict the flow of current in an electric circuit. 
Resistance is inherent in any electric wire, includ- 
ing those used for the transmission of electric 
power. Resistance in the wire is responsible for 
heating the wire as current flows through it and 
the subsequent power loss due to that heating. 

Restoration: The process of returning generators 
and transmission system elements and restoring 
load following an outage on the electric system. 

Safe Limits: System limits on quantities such as 
voltage or power flows such that if the system is 
operated within these limits it is secure and 
reliable. 

SCADA: Supervisory Control and Data Acquisi- 
tion system; a system of remote control and telem- 
etry used to monitor and control the electric 
system. 

Scheduling Coordinator: An entity certified by 
the ISO for the purpose of undertaking scheduling 
functions. 

Security: The ability of the electric system to with- 
stand sudden disturbances such as electric short 
circuits or unanticipated loss of system elements. 

Security Coordinator: An individual or organiza- 
tion that provides the security assessment and 
emergency operations coordination for a group of 
Control Areas. __ 

Short Circuit: A low resistance connection unin- 
tentionally made between points of an electrical 
circuit, which may result in current flow far above 
normal levels. 

Single Contingency: The sudden, unexpected fail- 
ure or outage of a system facility(s) or element(s) 
(generating unit, transmission line, transformer, 
etc.}. Elements removed from service as part of the 
operation of a remedial action scheme are consid- 
ered part of a single contingency. 

Special Protection System: An automatic protec- 
tion system designed to detect abnormal or prede- 
termined system conditions, and take corrective 
actions other than and/or in addition to the isola- 
tion of faulted components. 

Stability: The ability of an electric system to main- 
tain a state of equilibrium during normal and 
abnormal system conditions or disturbances. 
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Stability Limit: The maximum power flow possi- 
ble through a particular point in the system while 
maintaining stability in the entire system or the 
part of the system to which the stability limit 
refers. 

State Estimator: Computer software that lakes 
redundant measurements of quantities related to 
system state as input and provides an estimate of 
the system state (bus voltage phasors). It is used to 
confirm that the monitored electric power system 
is operating in a secure state by simulating the sys- 
tem both at the present time and one step ahead, 
for a particular network topology and loading con- 
dition. With the use of a state estimator and its 
associated contingency analysis software, system 
operators can review each critical contingency to 
determine whether each possible future state is 
within reliability limits. 

Station: A node in an electrical network where 
one or more elements axe connected. Examples 
include generating stations and substations. 

Substation: Facility equipment that switches, 
changes, or regulates electric voltage. 

Subtransmission: A functional or voltage classifi- 
cation relating to lines at voltage levels between 
69kV and 115kV. 

Supervisory Control and Data Acquisition 
(SCADA): See SCADA. 

Surge: A transient variation of current, voltage, or 
power flow in an electric circuit or across an elec- 
tric system. 

Surge Impedance Loading: The maximum 
amount of real power that can flow down a 
lossless transmission line such that the line does 
not require any VArs to support the flow. 

Switching Station: Facility equipment used to tie 
together two or more electric circuits through 
switches. The switches are selectively arranged to 
permit a circuit to be disconnected, or to change 
the electric connection between the circuits. 

Synchronize: The process of connecting two pre- 
viously separated alternating current apparatuses 
after matching frequency, voltage, phase angles, 
etc. (e.g„ paralleling a generator to the electric 
system). 

System: An interconnected combination of gener- 
ation, transmission, and distribution components 
comprising an electric utility and independent 


power producer! s) (IPP), or group of utilities and 
IPP(s). 

System Operator: An individual at an electric sys- 
tem control center whose responsibility it is to 
monitor and control that electric system in real 
time. 

System Reliability: A measure of an electric sys- 
tem’s ability to deliver uninterrupted service at 
the proper voltage and frequency. 

Thermal Limit: A power flow limit based on the 
possibility of damage by heat. Heating is caused by 
the electrical losses which are proportional to the 
square of the active power flow. More precisely, a 
thermal limit restricts the sum of the squares of 
active and reactive power. 

Tie-line: The physical connection (e.g. transmis- 
sion lines, transformers, switch gear, etc.) between 
two electric systems that permits the transfer of 
electric energy in one or both directions. 

Time Error: An accumulated time difference 
between Control Area system time and the time 
standard. Time error is caused by a deviation in 
Interconnection frequency from 60.0 Hertz. 

Time Error Correction: An offset to the Intercon- 
nection’s scheduled frequency to correct for the 
time error accumulated on electric clocks. 

Transfer Limit: The maximum amount of power 
that can be transferred in a reliable manner from 
one area to another over all transmission lines (or 
paths) between those areas under specified system 
conditions. 

Transformer: A device that operates on magnetic- 
principles to increase {step up) or decrease (step 
down) voltage. 

Transient Stability: The ability of an electric sys- 
tem to maintain synchronism between its parts 
when subjected to a disturbance of specified 
severity and to regain a state of equilibrium fol- 
lowing that disturbance. 

Transmission: An interconnected group of lines 
and associated equipment for the movement or 
transfer of electric energy between points of sup- 
ply and points at which it is transformed for deliv- 
ery to customers or is delivered to other electric 
systems. 

Transmission Loading Relief (TLR): A procedure 
used to manage congestion on the electric trans- 
mission system. 
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Transmission Margin: The difference between 
the maximum power flow a transmission line can 
handle and the amount that is currently flowing 
on the line. 

Transmission Operator: NERC-certified person 
responsible for monitoring and assessing local 
reliability conditions, who operates the transmis- 
sion facilities, and who executes switching orders 
in support of the Reliability Authority. 

Transmission Overload: A state where a transmis- 
sion line has exceeded either a normal or emer- 
gency rating of the electric conductor. 

Transmission Owner (TO) or Transmission Pro- 
vider: Any utility that owns, operates, or controls 
facilities used for the transmission of electric 
energy. 

Trip: The opening of a circuit breaker or breakers 
on an electric system, normally to electrically iso- 
late a particular element of the system to prevent it 
from being damaged by fault current or other 
potentially damaging conditions. See Line Trip for 
example. 

Voltage: The electrical force, or “pressure,” that 
causes current to flow in a circuit, measured in 
Volts. 


Voltage Collapse (decay): An event that occurs 
when an electric system does not have adequate 
reactive support to maintain voltage stability. 
Voltage Collapse may result in outage of system 
elements and may include interruption in service 
to customers. 

Voltage Control: The control of transmission volt- 
age through adjustments in generator reactive out- 
put and transformer taps, and by switching 
capacitors and inductors on the transmission and 
distribution systems. 

Voltage Limits: A hard limit above or below which 
is an undesirable operating condition. Normal 
limits are between 95 and 105 percent of the nomi- 
nal voltage at the bus under discussion. 

Voltage Reduction: A procedure designed to 
deliberately lower the voltage at a bus. It is often 
used as a means to reduce demand by lowering the 
customer’s voltage. 

Voltage Stability: The condition of an electric sys- 
tem in which the sustained voltage level is con- 
trollable and within predetermined limits. 

Watthour (Wh): A unit of measure of electrical 
energy equal to lwatt of power supplied to, or 
taken from, an electric circuit steadily for 1 hour. 
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Appendix D 

Transmittal Letters from the Three Working Groups 


Mr. James W. Glotfelty 
Director, Office of Electric Transmission 
and Distribution 
U.S. Department of Energy 
1000 Independence Avenue SW 
Washington, DC 20585 

Dr. Nawal Kamel 

Special Assistant to the Deputy Minister 

Natural Resources Canada 

580 Booth Street 

Ottawa, ON 

K1A0E4 


Dear Mr. Glotfelty and Dr. Kamel: 

Enclosed is the Interim Report of the Electric System Working Group (ESWG) supporting the United 
States - Canada Power System Outage Task Force. 

This report presents the results of an intensive and thorough investigation by a bi-national team of the 
causes of the blackout that occurred on August 14, 2003. The report was written largely by four 
members of the Working Group (Joe Eto, David Meyer, Alison Silverstein, and Tom Rusnov), with 
important assistance from many members of the Task Force’s investigative team. Other members of the 
ESWG reviewed the report in draft and provided valuable suggestions for its improvement Those 
members join us in this submittal and have signed on the attached page. Due to schedule conflicts, one 
member of the ESWG was not able to participate in the final review of the report and has not signed this 
transmittal letter for that reason. 


Sincerely, 


David H. Meyer 
Senior Advisor 
U S. Department 
of Energy 
Co-Chair, ESWG 



Senior Advisor 
Natural Resources 
Canada 

Co-Chair. ESWG 



Alison Silverstein 


Senior Energy Policy Advisor 
to the Chairman 
Federal Energy Regulatory 
Commission 
Co-Chair, ESWG 
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David Burpee, Director, 

Renewable and Elect ricai Energy Division 
Natural Resources Canada 



Connecticut Department of 
Public Utility Control 



Joseph Etta, Staff Scientist 
VS. p«paitment of Energy 
Lawrence Berkeley National Laboratory 
Consortium for Electric Reliability 
Technology Solutions 



Senior Vjcc President, Transmission 
New York Power Authority 



me Whitney, Policy Analyst / 

National Science and Technology Council 
U.S. Office of Science and Technology 
Policy 

Executive Office of the President 



(not able to participate in review) 

William D. McCarty, Chairman 
Indiana Utility Regulatory Commission 


David McFadden 

Chair, National Energy and Infrastructure 
Industry Group 

Gowlings, Lafleur, Henderson LLP 
Ontario 



David O’Brien, Commissioner 
Vermont Department of Public Service 



David O’Connor, Commissioner 

Div. of Energy Resources 

Massachusetts Office of Consumer Affairs 


And Business Regulation 



Alan Schriber, Chairman 
Ohio Public Utilities Commission 
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M Canadian Nuclear 
Safety Commission 

NUCLEAR REGULATORY COMMISSION 

•jUHHttWBAiMSMOT Pr**W*M»nd 

OMExaeuftraOfffeK 

CttUKUK 

November 5 , 2003 

PREDECISIONAL 

Mr. James Glotfelty 

Senior Policy Advisor 

Office of the Secretary 

U.S. Department of Energy 

1000 Independence Ave., Suite 7B-222 

Washington, DC 20585 

Dr. Nawal Kamel 

Special Assistant to the Deputy Minister 

Natural Resources Canada 

S80 Booth Street 

Ottawa. ON 

K1A0E4 

Dear Mr. Glotfelty and Dr. Kamei: 



CommiMioe eanadjome 
daeuratteucKaite 

Pr*tfdeftU *t 
pmrftm tlrtgwt* 


Enclosed for incorporation into the Task Force report is the interim phase-one report of 
the Nuclear Working Group supporting the United States - Canada Joint Power System Outage 
Task Force. The members of the Nuclear Working Group join us In this submittal and have 
signed the attached pages. This interim report is prededsional (not for public release) until 
you issue the Task Force interim report, and should be made available only to those Individuals 
needing this information to support the Task Force activities. 

Please provide any comments related to the Canadian nuclear plants to either Mr. Jim 
Blyth (613-995-2655; blythj@cnsc-ccsn.gc.ca), or Mark Dallaire (613-947-0957; 
dallairem@cnsc-ccsn.gc.ca). Comments on the U.S. nuclear plants should be directed to either 
Mr. Cornelius Holden (301-415-3036; dh@nrc.gov) or Mr John Boska (301-415-2901; 
jpb1@nrc.gov). 


Sincerely, 


U 

Nils J. Diaz / 

Chairman 

U.S. Nudear Regulatory Commission 
U.S. Co-chair, Nudear Working Group 




Linda J. Keen 
President and Chief Executive Officer 
Canadian Nudear Safely Commission 
Canadian Co-chair, Nudear Working Group 


Enclosures: Nudear Working Group Signature Pages (2) 

Nudear Working Group Interim Report Phase One 
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ccw/end: Mr. James Blyth 

Director General, Reactor Power Regulation 
Canadian Nuclear Safety Commission 

Mr. Samuel J. Collins 

Deputy Executive Director, Reactor Programs 
U.S. Nuclear Regulatory Commission 
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The members of the Nuclear Working Group hereby submit this report as input to the United 
States - Canada Joint Power System Outage Task; Force: 


Nils J. Diaz, Chairman J 
U.S. Nuclear Regulatory Commission 
Co-chair, Nuclear Working Group 


llumA- 


uel J. Collins, Deputy Executive Director 
for Reactor Programs 
U.S. Nuclear Regulatory Commission 




WIBam D. Magwood, IV, Director, Offics of 
Nuclear Energy, Science and Technology 
U.S. Department of Energy 


QJZ tr&ig../- 

Edward Wilds. Bureau of Air Management, 
Department of Environmental Protection 
(Connecticut) 



/ Jj Peter LariCChainnan, Public Service 
Commission (Michigan) 



tfnnor, Commissioner/bivision of 
sources, Office of Consumer 
Affairs and Business Regulation 
(Massachusetts) 



Frederick F. Butler, Commissioner. New 
' Jersey Board of Public Utilities (New Jersey) 


cay 1 ^ TOT. 


Paul Eddy, Power Systems Opei 
Specialist. Public Service Comi 
York) 



Dr. G. Ivan Maldonado, Associate Professor, 
Mechanical, Industrial and Nuclear 
Engineering; University of Cincinnati (Ohio) 



David J. Allard, CHP, Director, Bureau of 
Radiation Protection. Department of 
Environmental Protection (Pennsylvania) 



David O'Brien, Commissioner 
Department of Public Sendee (Vermont) 
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The members of the Nuclear Working Group hereby submit this report as input to the United 
States - Canada Joint Poster System Outage Task Force: 



Unda J. Keent 
President and Chief Executive Officer' 
Canadian Nuclear Safety Commission 
Co-chair, Nuctear Working Group 





^Jamecl 

jSSenerai, Directorate of Power 
Roador Regulation 
Canadian Nuclear Safety Commission 


eD£fr£ 

Ken Pereira 


Vice-President, Operations Branch 
Canadian Nuclear Safety Commission 



Chief Executive Officer 
Bruce Power 

(Representing the Province of Ontario) 


Dr. Roberrflwfsi 


kJ- Ms/ * 

rTWonfson ( 


Senior Advisor to the Deputy Minister 
Natural Resources Canada 
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Mr. James W. Glotfelty 
Director, Office of Electric Transmission 
and Distribution 
U.S. Department of Energy 
1000 Independence Avenue SW 
Washington, DC 20585 

Dr. Nawal Kamel 

Special Assistant to the Deputy Minister 

Natural Resources Canada 

580 Booth Street 

Ottawa, ON 

KIA0E4 

Dear Mr. Glotfelty and Dr. Kamel: 

Enclosed is the Interim Report of the Security Working Group (SWG) supporting the United 
States - Canada Power System Outage Task Force. 

The SWG Interim Report presents the results of the Working Group's analysis to date of the 
security aspects of the power outage that occurred on August 14, 2003. This report comprises 
input from public sector, private sector, and academic members of the SWG, with important 
assistance from many members of the Task Force’s investigative team. As co-chairs of the 
Security Working Group, we represent all members of the SWG in this submittal and have 
signed below. 


Sincerely, 



U.S. 

axxmit.swc 



Siatrt# wdtmtiiim**. 

>ri-y Coeocti 
Go*cmawnt OfcM* • 
C+Ck*k t sWG 
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Attachment 1 : 


U.S. -Canada Power System Outage Task Force SWG Steering Committee members: 


Bob Liscouski, Assistant Secretary for 
Infrastructure Protection, Department of Homeland 
Security (U.S. Government) (Co-Chair) 

William J.S. Elliott, Assistant Secretary to the 
Cabinet, Security and Intelligence, Privy Council 
Office (Government of Canada) (Co-Chair) 

U.S. Members 


Andy Purdy, Deputy Director, National Cyber Security 
Division, Department of Homeland Security 

Hal Hendershot, Acting Section Chief, Computer 
Intrusion Section, FBI 

Steve Schmidt, Section Chief, Special Technologies 
and Applications, FBI 

Kevin Kolevar, Senior Policy Advisor to the Secretary, 
DoE 

Simon Szykman, Senior Policy Analyst, U.S- Office of 
Science STechnoIogy Policy, White House 

Vincent DeRosa, Deputy Commissioner, Director of 
Homeland Security (Connecticut) 

Richard Swensen, Under-Secretary, Office of Public 
Safety and Homeland Security (Massachusetts) 

Colonel Michael C. McDaniel (Michigan) 


Sid Caspersen, Director, Office of Counter-Terrorism 
(New Jersey) 

James McMahon, Senior Advisor (New York) 

John Overly, Executive Director, Division of Homeland 
Security (Ohio) 

Arthur Stephens, Deputy Secretary for Information 
Technology, (Pennsylvania) 

Kerry L. Sleeper, Commissioner, Public Safety 
(Vermont) 

Canada Members 

James Harlick, Assistant Deputy Minister, Office of 
Critical Infrastructure Protection and Emergency 
Preparedness 

Michael Devaney, Deputy Chief, Information 
Technology Security Communications Security 
Establishment 

Peter MacAulay, Officer, Technological Crime Brandi 
of the Royal Canadian Mounted Police 

Gary Anderson, Chief, Counter-Intelligence - Global, 
Canadian Security Intelligence Service 

Dr. James Young, Commissioner of Public Security, 
Ontario Ministry of Public Safety and Security 
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REVIVING THE ELECTRICITY SECTOR 

Key Findings of the National Commission on Energy Policy 


The National Commission on Energy Policy was founded in 2002 by the William and Flora 
Hewlett Foundation, and its partners — the Pew Charitable Trusts, the John D. and Catherine T. 
MacArthur Foundation, the David and Lucile Packard Foundation and the Energy Foundation. It 
is currently developing comprehensive recommendations for long-term national energy policy to 
be released in December 2004. 
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John P. Holdren 
Co-Chair 

Teresa and John Heinz Professor of Environmental Policy, 
Harvard University 

William K. Reilly 
Co-Chair 

President and CEO, Aqua International Partners; former 
Administrator of the Environmental Protection Agency 

John W. Rowe 
Co-Chair 

Chairman and CEO, Exelon Corporation 
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Congressional Chair 

Senior Advisor. Lexccon, Inc.; Senior Policy Advisor, Van Ness 
Feldman; Former U.S. Representative, IN 

Marilyn Brown 

Director, Energy Efficiency and Renewable Energy Program, Oak 
Ridge National Labs 

Ralph Cavanagh 

Senior Attorney & Co-Director, Energy Program, Natural 
Resources Defense Council 
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Rodney Ellis 

State Senator, Texas 


Leo Gerard 

International President, United Steelworkers of America 

F. Henry Habicht 
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Professor of Economics and Director of MIT Center for Energy 
and Environmental Policy Research, Massachusetts Institute of 
Technology 

Andrew Lundquist 
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Sharon L. Nelson 
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General’s Office; Chair, Board of Directors, Consumers Union 
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Electric- industry restructuring has derailed. The massive blackout of August 14, 2003 
certainly was not needed to underscore the point, but it adds urgency to the effort to find 
solutions. Wholesale markets continue to evolve slowly and erratically but are impeded by state- 
federal conflict, regulatory and legislative uncertainty, malfeasance, poor credit and outright 
collapses, of which Enron is only the most notorious. FERC’s efforts to promote more efficient 
markets through regional transmission organizations and a wholesale market platform offer 
promise, but have generated confusion and opposition. In the last five years, increased 
generation competition has elicited more than 100,000 megawatts of gas- fired peaking and 
baseload capacity, which has contributed both to a period of relatively low wholesale prices in 
many regions and increased exposure to gas price volatility across the system. But competitors’ 
losses have created substantial uncertainty about how quickly and on what terms capital markets 
will support additional investment throughout this sector. Indeed, investment in all categories of 
electricity infrastructure is down significantly, in part because of surplus capacity conditions in 
certain regions, but also because of uncertainty concerning which entities have the responsibility 
for identifying and making investments in the transmission and distribution networks, and 
uncertainties about how the associated costs will be recovered. A challenge in reviving these 
capital flows is to clarify prospects for cost recovery and reward: for example, when and on 
what terms will distribution utilities have the ability to enter into long-term contracts with 
generation service providers; how will distribution utility responsibilities interact with the 
opportunities created for competitive retail suppliers in states with retail competition; who has 
the responsibility for identifying needed enhancements to the transmission network; how will 
they be paid for securing them; and who will pay? The August 2003 blackout is a reminder of 
how much hinges on finding practical answers promptly. 

Individual states have varied greatly in their willingness to introduce retail electricity 
competition, and their enthusiasm for federal policies designed to promote wholesale 
competition. Even in states that have opted for retail competition, efforts to expand it have 
generally halted in the wake of the Enron collapse and the California disaster. Large industrial 
customers often have benefited from retail competition, effectively exercising their ability to 
“buy wholesale” whenever prices are lower than the “safety net” of regulated rates that such 
states typically provide. These customers seldom seek “value-added” electricity service; rather, 
they seek the cheapest commodity prices and the shortest contractual commitments. Large 
customers contend that their continued exposure to some utility charges impedes the further 
development of these markets. Utilities contend that continued safety nets for the industrials 
have the same effect. 

Small customers sometimes have benefited from rate guarantees in restructuring 
legislation, but they have received little direct benefit from retail competition itself. Because the 
pocketbook advantages have been insubstantial, many consumers find the choices associated 
with retail competition to be more of an annoyance than an advancement over past service 
offerings. Retail marketers have lost some billions in capital without developing a profitable, 
sustainable and distinct value-added product, although a few pioneers have made intriguing 
efforts to market products based on environmentally preferred generation sources. 


3 


336 



National 
Commission 
on Energy 
Policy 


1616 H Street, NW 
6™ floor 

W ashingtok DC 20006 
P 202-637-0400 
F 202-637-9220 

WWW. ENERGYCOMMJSS10N.ORG 


At the same time, it is often unclear who is responsible for assembling a diversified mix 
of short- and long-term resource commitments and other risk management tools, in order to 
sustain the economical and reliable electricity services that a healthy economy requires. 
Competitive models assume that decisions by market participants will replace resource planning 
by utilities or regulators. In practice, however, competitive models have retained — whether in 
utilities, in regional transmission organizations or in the states themselves — some residual 
responsibility for ensuring that electricity supplies remain adequate. In some restructuring 
models, customers unwilling or unable to choose a supplier have been provided with default 
options that influence the evolution of the market. These “carrier of last resort” options also fail 
to address either the real relationships between wholesale and retail markets or the complex 
issues involved in resource planning. Indeed, in California, events evolved such that from 1998- 
2001 utilities were required to supply power to retail consumers at frozen rates after losing the 
ability to enter into forward contracts for the power that they were obligated to deliver.. 

In states with traditional regulatory regimes, the regulated utilities that provide most 
resource procurement and management services generally do so based on longstanding cost 
recovery principles, with abundant downside risk and little or no prospect of gain regardless of 
the quality of their performance. In states with retail competition, the retail suppliers view long- 
term procurement by distribution companies as unfair competition, and the distribution 
companies face potential stranded cost problems or prudence reviews from regulators if they do 
make resource commitments. Yet failures to make such commitments may force expensive 
purchases in volatile short-term markets, which may result in adverse treatment by regulators. 

Even in states that do not have retail competition programs, the threat of their 
introduction and stranded costs deters long-term commitments by investor owned utilities, even 
as risks of regulatory review make the alternative of short-term purchases look dangerous for 
utility shareholders. Utilities, regulators and wholesale suppliers alike are struggling with how 
states can regulate retail electric service provided by companies that operate in wholesale power 
markets that cross state lines. All parties are stuck between uncertain regulatory regimes, with 
no assurance about the rules that will determine commercial survival and success. 

Finally, the electric industry's environmental footprint is significant, and a wide range of 
technologies and technology vintages means widely varying emissions and other impacts from 
the competitors for generation and grid investments. While there have been important reductions 
in some power generation pollutants, the sector’s greenhouse gas emissions have been increasing 
more rapidly than those of the rest of the economy. National policy on greenhouse gases and 
other key pollutants remains uncertain, and states are beginning to act on their own initiative to 
reduce these emissions. This continuing policy struggle and growing jurisdictional tension 
creates an additional source of uncertainty for the industry, with serious implications for different 
technology options, electricity service costs, and environmental consequences of electricity 
production and transmission. 

Overcoming these formidable challenges requires a balancing of the extent to which 
electricity is a commodity and a public service. Also needed are an evaluation of the benefits of 
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competition and other mechanisms for achieving public utility goals, and an integration of the 
flexibility of spot markets with the increased certainty of planning. To complicate matters 
further, key decision-makers at different levels of government are at odds over who should make 
these decisions and how. 

The Commission sees an urgent need to address and help resolve these issues. Both state 
and federal regulators have vital and complementary roles to play in providing consumers with 
the benefits of properly structured electricity markets. Within the context of pending regulatory 
and legislative proposals at both federal and state levels, we aim to help define those roles and to 
offer a vision for revitalizing both wholesale electricity markets and broader electricity-resource 
procurement and management responsibilities. In the process, we will explore the most 
promising ways to encourage appropriate electricity- resource and grid investments. 

Absent the prospect of retail competition, of course, this would be an easier problem to 
solve. 1 Under regulatory oversight, distribution companies could have relatively well defined 
retail supply obligations, met through some combination of wholesale contracts, demand-side 
investments and ownership of generation assets. A crucial issue, then, is how to think about 
retail competition: if we are going to have it, how can we make it work and speed the transition? 
If instead we prefer to reject retail competition, how do we make that decision credible enough 
for distribution companies and others to take it to the bank? If different states and regions 
choose different models, how will those variations intersect with national policies that favor 
more standardization for wholesale power markets and the role of transmission systems (and 
regional transmission organizations) in enabling them? 

Finally, no assessment of our electricity challenges would be complete without careful 
attention to the system’s vulnerability to terrorist attack. Much of the electricity infrastructure is 
in private hands, so protecting that infrastructure will require a strong government-private sector 
partnership. Although the grid is more resilient than many may appreciate, some equipment has 
long replacement lead-times and constant vigilance is essential to guard against potential 
disruption of the grid control systems. Attacks could be either cyber-based or physical, or some 
combination of the two. These issues deserve, and are getting sustained attention from, 
institutions like the Department of Homeland Security, the Department of Energy, the Federal 
Energy Regulatory Commission, the National Academies, numerous state agencies, and the 
North American Electric Reliability Council. The Commission’s recommendations below reflect 
and reinforce their vital work. At the same time, although it has not been linked to sabotage, the 
August 2003 blackout is a reminder that reliability concerns demand strong enforcement of 
mandatory reliability standards as a replacement for today’s overburdened voluntary system; the 
Commission adds its voice to those who have been urging Congress to take specific action here. 


1 Other important questions include whether ioad serving entities should be generation owners or not, and whether 
distribution companies will retain ownership of transmission. A central and still unresolved issue is whether 
wholesale competition can flourish (or flourish enough) in a world that includes vertically integrated utilities ( i.e. . 
utilities that own generation, transmission and distribution assets). 
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THE COMMISSION’S INITIAL RECOMMENDATIONS 

The Commission supports an energy policy that recognizes both the quarter-century march 
toward increased competition in electric generation and wholesale markets and the value of 
traditional modes of regulatory authority. A fundamental assumption is that the wholesale 
electricity business is largely a competitive commodity business. So too is the retail supply of 
electricity to very large customers, including industrial customers and some national chains. 
However, the retail supply of electricity to other customers is, for the foreseeable future, likely to 
remain a service- oriented business with major public policy implications. In this context, 
electric distribution and transmission companies have both special opportunities and special 
obligations. As the federal go vemment and the states attempt to resolve the tensions inherent in 
promoting competition and customer choice, multiple paths may be found to widely shared 
equity, environmental and economic objectives. 

These recommendations constitute a framework that the Commission presents as a prototype 
for progress in accommodating diverse needs and goals: 

FOR STATE REGULATORS AND BOARDS OF CONSUMER-OWNED UTILITIES: 

1 . Retail distribution should remain a responsibility of utilities under state and local 
regulation, along with electric energy resource portfolio management for residential and 
small business customers (and any larger customers who choose regulated portfolio 
services). 2 If customers, especially large users of electricity, are permitted to opt out of 
regulated portfolio service and to make their own choices in retail electric markets, they 
should be allowed to return to regulated service only on terms that hold harmless other 
customers and the regulated portfolio manager. For small customers in states that opt for 
retail electricity competition, schedules should be established to allow for orderly 
provision of retail choice opportunities in phases across service territories, with all small 
customers having opportunities to choose alternative portfolio managers no less than 
every five years. 

2. Large customers who choose regulated portfolio service should be required to execute 
long-term contracts with the utility portfolio manager. Large customers who do not opt 
for regulated portfolio services should make their own way in the competitive retail 
markets. 

3. State regulators and boards of consumer-owned utilities need to focus more on incentives 
for good portfolio management service. Options include systems of performance- based 
regulation for regulated portfolio management (and other) services provided by retail 
distribution companies, based on objective benchmarks, and incentives for managers and 


2 As indicated in the introductory section, by “electric resource portfolio management” the Commission means 
“assembling a diversified mix of short- and long-term resource commitments and other risk management tools, in 
order to sustain the economical and reliable electricity services that a healthy economy requires.” 
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(where applicable) shareholders reflecting reasonable measures of net benefits delivered 
to customers. Regulated distribution companies can be compensated independently of 
increased electricity sales (for example, utilities’ fixed-cost recovery can be made 
independent of retail electricity use, through the mechanism of small periodic upward or 
downward adjustments in distribution rates). For purposes of meeting portfolio 
management responsibilities, reliable load reductions and reliable generation, including 
small-scale “distributed” generation at or near load centers, should all be investment 
candidates. The goal should be to hold regulated portfolio managers accountable but 
also to avoid complex regulatory review processes. 

FOR THE FEDERAL ENERGY REGULATORY COMMISSION: 

4. The Commission supports FERC’s efforts to ensure nondiscriminatory transmission 
operations and nondiscriminatory access to grids and wholesale markets, with appropriate 
deference to the needs of states that have not adopted retail competition and states’ 
crucial role in ensuring resource adequacy. Congress should authorize the extension of 
those requirements to all transmission regardless of who owns it. The Commission 
believes that these policies are needed to revitalize competitive wholesale electricity 
markets. Wholesale market participants should win or lose based on their ability to 
maximize operating efficiencies under a deregulated price regime untainted by exercises 
of market power 

5. To improve system security and reliability, the national electricity system needs to 
maintain dispersed and well guarded stockpiles of critical equipment with long 
replacement lead-times, and to standardize such equipment wherever feasible. Prompt 
attention should also be given to ensuring the security of Supervisory Control and Data 
Acquisition (SCADA) systems. Also important are joint government-private sector 
efforts to complete the studies necessary to mitigate the effects of and accelerate recovery 
from terrorist attacks. The costs of these efforts, and other costs involved in improving 
grid security, should be shared system- wide on a competitively neutral basis, through 
uniform charges on transmission use administered by the FERC. In view of the national 
importance of this objective and its relatively modest cost when spread across the 
nation’s electrical grid, Congress should provide for the collection of these charges 
notwithstanding state- mandated retail rate freezes. 

FOR CONGRESS: 

6. Both societal and generation- sector interests would benefit substantially from more 
coordination and greater certainty regarding targets and timetables for achieving long- 
term environmental objectives. Accordingly, for all categories of power plant emissions 
that it considers appropriate subjects of regulation, Congress should establish an 
integrated regulatory structure that (1) establishes a firm multi-year schedule of phased 
emission reductions that accommodates both environmental and system reliability needs; 
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and (2) uses market- based mechanisms to the maximum extent feasible to minimize 
compliance costs and encourage innovation. 

7. Congress also should tighten energy efficiency standards wherever practicable and cost- 
effective, in view of the substantial environmental and economic costs associated with 
unnecessary use of energy. 

8. The August 2003 blackout was a terrible reminder that the system of voluntary 
compliance with non-binding reliability rules for electricity grids is breaking down across 
North America. Congress should approve widely supported proposals to make such 
reliability rules mandatory and enforceable, when promulgated by a FERC-approved 
North American electric reliability organization working with regional bodies 
accountable to all owners, operators and users of bulk power systems, and with ultimate 
oversight responsibility vested in the Federal Energy Regulatory Commission. See also 
item 5 above. 

FOR ALL DECISION-MAKERS: 

9. Wholesale electric markets work best when they are liquid and transparent, for real time, 
day ahead and long-term products. The Commission supports FERC’s proposals for real- 
time and day-ahead wholesale markets, along with state- level policies designed to ensure 
that such price signals are much more effectively communicated to large customers or 
aggregators at the retail level. More transparency for spot market prices and volumes of 
electricity trading, with reporting as close as possible to real time, are urgent priorities. 

10. While the Commission is encouraged by the emergence of innovative technological 
solutions to transmission reliability and congestion problems, we agree that inadequate 
investment in transmission infrastructure is a significant and growing national problem. 
Transmission owners should be challenged to identify and consider all potentially cost- 
effective solutions to congestion and reliability problems, including targeted demand 
reductions, replacements of existing facilities with better equipment and new technology, 
and new facilities. No single solution will suffice; we need a portfolio that includes using 
new technology as well as constructing new transmission lines. FERC should also clarify 
which entities are responsible for identifying and making transmission investments, how 
they will be paid, and who will pay the associated costs. Options for encouraging cost- 
effective investment include higher rates of return for approved measures, increased 
certainty of recovery, and performance-based rewards that share system savings between 
shareholders and users. In addition, confusion and controversies created by FERC’s 
interest in merchant transmission investment, and ambiguities about the practical 
meaning and application of the “participant funding” concept, are discouraging 
investment and must be clarified and resolved. 

1 1 . Congress, FERC and state regulators should encourage interconnected electricity systems 
to undertake more regional resource and grid enhancement planning. 
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12, Urgent action is needed to revive the electricity sector’s research and development 
investments, always low by any reasonable standard and down by more than three- 
fourths in real terms over the past two decades. The Commission favors supplementing 
the federal budgetary contribution with a combination of federal tax incentives and state- 
approved utility investments, recovered as small charges on electric distribution, such as 
those that created the Electric Power Research Institute. 
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The following Commissioners join in support of the recommendations stated herein: 

Dr. Marilyn Brown 

Director, Energy Efficiency and Renewable Energy Program, Oak Ridge National Laboratory 

Ralph Cavanagh 

Senior Attorney & Co-Director, Energy Program, Natural Resources Defense Council 

Archie W. Dunham 

Chairman, ConocoPhiilips 

Rodney Ellis 

State Senator, Texas 

F. Henry Habicht 

CEO, Global Environment & Technology Foundation 

Dr. John P. Holdren 

Teresa and John Heinz Professor of Environmental Policy, Harvard Un iversity 

Dr. Paul L. Joskow 

Professor of Economics and Director of MIT Center for Energy and Environmental Policy Research, Massachusetts 
Institute of Technology 

Andrew Lundquist 

President, The Lundquist Group 

Dr. Mario J. Molina 

Institute Professor, Massachusetts Institute of Technology 

Sharon Nelson* 

Chief, Consumer Protection Division, Washington Attorney General’s Office; Chair, Board of Directors, Consumers 
Union 

William K. Reilly 

President and CEO, Aqua International Partners; Former Administrator of the Environmental Protection Agency 

John W. Rowe 

Chairman and CEO, Exelon Corporation 

Phillip R. Sharp 

Senior Advisor, Lexecon, Inc.; Senior Policy Advisor, Van Ness Feldman; Former U.S. Representative, IN 

Linda Stuntz 

Stuntz, Davis & Staffier 

Susan Tierney 

Managing Principal, The Analysis Group 

R. James Woolsey 

Vice President, Booz, Allen, Hamilton; former Director of the Central Intelligence Agency 

Dr. Martin B. Zimmerman 

Group Vice President, Corporate Affairs, Ford Motor Company 
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* Special Concurrence from Commissioner Sharon Nelson 

Senior Assistant Attorney General; Chief, Consumer Protection Division, Washington Attorney 
General’s Office 

I respectfully concur. The commission's statement is a committee work product on a complex 
subject. Like all such collegial efforts, the paper suffers from compromises and code words. I 
worry that some of the less obvious compromises will be used in Congressional and other policy 
debates for purposes not intended by any commissioner. However, the report contains many 
meaningful recommendations which advance the policy debate currently underway in the 
nation's capital and in other policy venues. 

The electricity sector provides an essential infrastructure for assuring the public safety, health 
and welfare. This report recognizes this practical reality and the significant need to re-establish 
some semblance of predictability for the electricity sector. It also encourages important efforts to 
address national security concerns, promotes coordinated regulation of all power plant emissions, 
encourages greater emphasis on energy efficiency and supports much needed technology R & D. 
The report also recognizes that other values besides market values still vitally affect the 
electricity industry and are affected by it. For these reasons, I support the report, despite 
the concerns described below. 

I reside in a region of the country which has suffered from "market designs" we sought to avoid. 
In my view, markets are not designed. They may evolve, they may be influenced by public 
policy but they are not the product of legislative or regulatory mandates. This report should be 
understood as merely early input on a still fitfully evolving "competition" policy in electricity. 

The report refers to the nation's quarter century trend toward competition in markets formerly 
viewed as de jure or de facto monopolies and implies that this forward march should not be 
interrupted by "inappropriate" state retail competition policies. In my opinion, there were good 
reasons for the electricity industry to be the last of the network industries to experience 
"restructuring." As opposed to the transportation, banking, or telecommunications industries, the 
preconditions which characterized the other sectors' reformations (such as ease of access to 
capital markets, freedom of entry, well understood rules about interconnection) did not exist in 
the vertically integrated electricity industry. Indeed, one major difference here is the ownership 
structure of the industry. As opposed to the natural gas industry or wireline telecom industry, the 
electricity industry is characterized by suppliers which are not investor owned. For example, 
in Washington State, two thirds of retail electricity sales are provided by customer 
or municipally owned providers. Traditional institutional oversight for this complex industry is 
not the same as the parallel natural gas or telecom markets “enjoyed,” making legislative and 
regulatory initiatives even more complicated. The phrase “ensuring a level playing field” is a 
hackneyed one, but this common sense goal is practically not achievable for the entire electricity 
industry in the nation's current electoral-political environment. In my view, in 2003, the nation 
needs a far more thoughtful analysis of why the experiments in Pennsylvania, California, the 
United Kingdom and Texas are succeeding or failing. Once we draw some lessons from 
empirical studies, then maybe some more far reaching and sensible policy reforms would flow. 


11 


344 



National 
Commission 
on Energy 
Policy 


1 6 ! 6 H Street, N W 
6 "'FLOOR 

W LSHWGTON DC 20006 
P 202-637-0000 
F 202-637-0220 

WWW. ENERGYCOMMISSION. ORG 


I have specific concerns with recommendations 9 and 10. Iam concerned that they provide too 
much deference to the Federal Energy Regulatory Commission which at this point does not 
manifest the institutional competence to warrant such trust, do not recognize regional 
differences or operational differences between thermal and hydro-electric systems, and are at 
once vague and overly prescriptive. 

Despite these concerns, I support a significant majority of the paper's recommendations. The 
debate over the future direction of our nation's electricity system is fundamentally stymied. The 
hard work and significant agreements reached by our expert and diverse Commission causes me 
to conclude that the overall report advances the national policy debate. For this reason 1 concur. 


SPECIAL NOTICE - Leo Gerard, President of the United Steelworkers of America (USWA) 

Leo Gerard joined the National Commission on Energy Policy after the bulk of work on this 
paper was completed. As a result, Mr. Gerard takes no position on the paper's content or 
recommendations. The issues raised in this paper are of significant interest to Mr. Gerard and to 
the USWA. Mr. Gerard will work actively in the coming months to ensure that the NCEP enjoys 
the benefit of the labor perspective when crafting its final recommendations. 
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CONTRIBUTING FACTORS TO THE SERIES OF 
OUTAGES ON AUGUST 14, 2003 

White Paper - September 23, 2003 
Objective of the Study 

At the request of FirstEnergy Corp., EPRJ performed an independent assessment of 
factors contributing to the series of outages on August 14, 2003 , which affected millions 
of electricity customers in the United States and Canada. This white paper, prepared by 
EPRI, describes that assessment. Over a period of three weeks, the study focused on data 
collection and analysis in two areas: (1) systemic factors that may have made the power 
system in this region particularly vulnerable to outage, and (2) specific operational and 
performance factors that may have helped precipitate the August 14 outage. It is 
important to recognize that, because this study was performed in a such a short timeframe 
and was based on data from only FirstEnergy Corp. and public sources these are only 
preliminary observations by the EPRI team. Such observations, however, may be helpful 
to the official U.S./Canada Power Outage Task Force in identifying areas deserving 
additional data collection and investigation and larger systemic issues requiring more 
thorough analyses to determine any appropriate corrective action. 

Key Findings 

A number of systemic issues may have contributed to the chain of events leading up to 
the August 14 outage. These need to be further explored with an eye toward preventing 
large-scale regional outages in the future. These include the following: 

• Lack of understanding of VAR reserves in the region and the adjacent regions, 
coupled with the impact of lack of financial incentives for entities to provide 
adequate VAR support. (VAR is shorthand for reactive power, which is the 
additional power required for maintaining voltage stability when serving certain 
kinds of load, such as motors, air conditioning, and fluorescent lights.) To 
understand the overall outage and develop appropriate mitigation 
recommendations, it is important for the U.S. / Canada Power Outage Task Force 
to devote more resources to the collection of all relevant information and 
incorporate it into models and analyses. 

• Insufficient “visibility” of power flow conditions over the entire region, coupled 
with inadequate coordination, control and communication of the power system on 
a regional basis. 

• Insufficient understanding of new power flow patterns caused by increased 
wholesale power transfers resulting from industry restructuring. 

• Lack of real-time regional and interconnection-wide power flow models for 
anticipating changing flow patterns and the formation of new bottlenecks. 

In addition, a number of specific operational issues that occurred on August 14 may have 
contributed to the outages and are deserving of further analysis. These include the following: 
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• Power flows from merchant plants in southern Ohio, providing power to the north, 
heavier than on prior days without adequate region-wide VAR support to 
maintain voltage stability in the region. 

• An unexplained cessation of alarms at FirstEnergy between 2:35 PM and 3:30 PM 
EDT, August 14. The alarm failure may have handicapped the operators at 
FirstEnergy. However, other control areas also reported alarm performance 
problems as well as data communications problems, which may have hampered 
Internet access and energy management systems (EMS) hardware and software 
performances. These may be systemic issues worthy of further review along with 
the communication and coordination among regional reliability coordinators. 

• An unusual number of unexplained “spurious” trips of power lines before they 
were fully loaded to their thermal limit. Further investigation is needed to 
determine all causes for the trips, e.g., if they sagged into trees and shorted out or 
if the lines tripped for overloads, under-voltage, over-excitation, under or over- 
frequency, etc., to determine any improvements on protection philosophy and 
practices, including wide-area protection coordination. 

Study Approach 

In identifying the contributing factors for the disturbances that occurred after 4 PM (all 
time is EDT) on August 14, 2003, the EPRi study team decided to examine the system 
conditions starting from mid-morning of the day of the disturbance. 

Collect Data and Synchronize Time Stamps 

The study process involved three steps. The first was to synchronize the time stamps of 
the different sources of recorded data in order to precisely determine the timings of all the 
disturbances recorded by FirstEnergy. The majority of data for this analysis was data 
archived by the MISO energy management system (EMS) in 30-second time increments. 
Archived data from the FirstEnergy EMS is primarily limited to integrated hourly 
average data, although digital fault relay (DFR) and other information used is captured at 
sub-cycle resolution. Actual voltages at a number of FirstEnergy busses, and MW and 
MVAR flows on a number of lines, were collected and correlated. Data in the Cleveland 
area was limited. 1 Data from FirstEnergy concerning the adjacent operating regions was 
limited. Because the sequence of events on August 14 occurred over a wide area of the 
Eastern Interconnection, the lack of data about the grid beyond the FirstEnergy service 
areas limited the team’s ability to make definitive conclusions. 

Conduct Analysis to Understand Events 

The second step, after collecting data, was an independent data review and analysis to 
obtain a better understanding of the sequence of events. This effort benefited in part from 
the timelines developed by various parties which were released to the public. The work 
involved reviewing the geographical locations and the timing of events and analyzing 


! The MISO EMS directly receives real-time RTU data from FirstEnergy directly through the NERC 
Interregional Security Network (1SN), and should have these data in its archive. 
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available data for indications of whether the events might have been either related or 
purely random and independent. The team reviewed four primary areas of data, (a) MW, 
MVAR and % duty line flows, (b) voltages, (c) system frequency, and (d) alarm system 
reports and operator telephone logs. 

Deliver Insights and Observations 

The third part of this study was to synthesize the preliminary observations from the 
independent review and analysis into a set of potential contributing factors that may help 
explain the series of events on August 14, 2003. This white paper presents some of these 
key insights and observations. The information that follows is organized temporally as 
the events unfolded. 

Initial Conditions Preceding 12:00 Noon Eastern Daylight Time 

As noted in the last part of the paper, the data received suggests that the events of August 
14 were not independent random coincidences. Therefore system conditions and the 
sequence of events may be viewed in that context. 

Typical Summer Weather - No Serious Adverse Conditions (on the 
surface) 

On the surface, August 14, 2003 was a typical summer weekday. There were no extreme 
weather conditions. Certain regions experienced relatively high temperatures while other 
regions were cooler. Temperatures were moderately high in the Eastern U.S. The 
temperature in Canton, Ohio, peaked at 87° F at 3 PM (humidity 48%, and heat index of 
89° F, cloud coverage 30%). Ontario was facing rising temperature over the previous 
two days. The high temperature in Toronto was 88° F, following a high of 85° on August 
13. 

On average, Ontario was a net power exporter throughout the daytime periods in July 
2003. 2 Between August 1 and August 13, Ontario exported at night and imported less 
than 500 MW during the daytime. On August 14, from the same data source, Ontario’s 
scheduled import was about 2000 MW at 1 p.m. EDT and increased to about 2300 MW 
by 3 PM EDT. 

As a result of these conditions, the wholesale power transfer pattern on August 14, 2003 
in Ohio was essentially from west to east and from south to north. This is in contrast to 
the typical flow pattern, which is from north to south and west to east. 

Transmission Lines and Generators out of Service on August 14 

A number of generating units and transmission facilities were out of service for 
scheduled maintenance on August 14 in the Eastern Interconnection (see Table 1 for a 
partial list of transmission lines and generators which were out of service on August 14). 


2 Based on NERC TagNet data. 
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Table 1 - Partial List of Transmission and Generation Facilities 
Out of Service for Scheduled Maintenances Near the Affected Areas 


Transmission Lines and Equipment 

Generators (Capacity, MW) 

FirstEnergy: 

• Fox Q- 1 -FX-C 1 3 8 Cap Bkr 

• Eastlake 62-EL-T 345/138 kV 
transformer 

• 

FirstEnergy: 

• Davis Besse (883) 

• Eastlake unit 4 (240) 

• Sammis unit3(180) 

IMO: 

• B3N (1 of 4 ties to Ontario) 

• J5D (phase shifter capability) 

• L5 1 D (phase shifter) 

IMO: 

• Atikokan (200) 

• Pickering unit 7 (500) 

• Nanticoke unit 1 (500) 

• Lakeview 1 (300) 

• Lakeview 2 (300) 

PJM: 

• Linden - Goethals 

• Warren - Falconer 1 1 5 kV line 

• Plymouth - Whitpain (220- 1 4) 230 
kV line 

• Bath County - Valley 500kV line 

• Juniata #1 Transformer 500/230 kV 

• Dickerson-Dickerson H 23103 & 
23104 230 kV lines 

PJM 

• Keystone 2 (850) 

• Hudson 1 (383) 

AEP 

(Not available) 

AEP 

• Cook 2 (1060) 

• Tanners Creek 4 (500) 

• Glen Lyn 6 (235) 

• Conesville 5(375) 

• Gavin 2 ( 1300) but switched back 
@ 12:05 PM 


In this day-ahead look, no apparent inadequacy existed between the MW generating 
capacity and loads, and between generating schedules and transmission capabilities in the 
Eastern Interconnection. However, unavailable generating units imply a decreased 
reactive capability to regulate system voltages, and perhaps reduced dynamic reactive 
reserves. The industry generally lacks timely, accurate data on the amount of dynamic 
and static reactive capacities on line over a wide area at any time, including the amount 
of reactive reserve available in the different parts of the Eastern Interconnection. 
Consequently, the EPRI project team could make no definitive statement about the 
available voltage support capability in the entire Eastern Interconnection on August 14, 
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2003, However, as shown in the analysis later in this paper, this subject deserves further 
investigation. 

Independent Power Producers 

The operation of independent power producers in Ohio and Indiana on August 14, 2003 
may have been one contributing factor in the sequence of events. Based on public 
information at the Ohio Power Siting Board (OPSB), one new 850 MW combined cycle 
merchant power plant at Waterford, Ohio began commercial operation on August 6, 2003. 
A second merchant plant (620 MW), also at Waterford, Washington County, Ohio, was 
in operation since about September 2002. A third merchant plant. Hanging Rock located 
in fronton, Ohio, had a second combined cycle unit ready for commercial operation on 
July 14, 2003 after its first unit was declared to the OPSB as being ready for commercial 
operation on June 10, 2003. Its total capacity is 1240 MW. Altogether, the total 
generating capacity from these three merchants, all located in the southeastern part of 
Ohio, was 27 1 0 MW, of which 1470 MW was new generation. The reactive support and 
voltage control capability available from these plants, and other generation, is unknown 
at this time. From the NERC TagNet data, there appeared to be no scheduled sales from 
the Waterford plant until August 14, 2003 when it was scheduled to start generating at 5 
AM EDT with output reaching about 800 MW through most of the day (see Figure 1). 

In Indiana, on August 14, 2003 at 13:15 EDT, the merchant plant at Wheatland was 
involved with an overload and system voltage problem near Gibson, Indiana, according 
to the MISO operator phone transcripts 3 . In addition to these merchant plants, there may 
have been other relatively new merchant plants in various parts of the Eastern 
Interconnection operating on that day. The scheduling and actual operation of these 
merchant plants, including automatic voltage control and VAR support, is generally not 
well known by grid operators because of commercial sensitivities and FERC regulation, 
unless the merchant plants are under direct contract and legal obligation to the connected 
grid to provide such ancillary services. This lack of information on the independent 
power producers was likely a contributing factor to the speed of grid operator response to 
emergencies on that day. From analyzing the NERC TagNet data, the schedules of the 
IPPs in southeastern Ohio and those of certain control areas on August 14, 2003 were 
computed. Figure 1 shows these schedules. Readers are cautioned against concluding that 
these were the actual generation or net interchanges that took place. Data on the actual 
generation and net interchanges should be collected from all control areas and all IPPs in 
the Eastern Interconnection before drawing such conclusions. 


3 Available from the House Committee on Energy and Commerce web site. 
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August 14, 2003 Scheduled Net Exports {MW) According to NERC TagNet Data 



Figure 1 - Net Schedules of IPPs in southeastern Ohio, AGP, OVEC and Ontario (IMO) 
on August 14, 2003 From NERC TagNet Data (DELO = Hanging Rock, DEWO = 
Washington, AEVVC = Wheatland) 


NERC uses Central Standard Time (CST) for the 1DC and for E-tags even during 
daylight saving time. Thus the original data in Figure 1 were reported in CST. 


As shown in Figure 1, Ontario (IMO) was importing on August 14, 2003 and its 
scheduled import continued to increase between 12 noon and 3 PM EDT. The total 
scheduled output of the three merchant plants in southeastern Ohio ramped up rapidly 
starting at 8 AM and peaked at 2600 MW between 1 PM and 3 PM EDT. From about 1 1 
AM, AEP’s schedule showed a rapid reduction in its export, by an amount of about 2400 
MW. 


Sequence of Events on August 14, 2003 

As we note later, the issue of reactive reserve margin in all or parts of the Eastern 
interconnection deserves further investigation. This could have contributed to voltage 
support in a wide area not being efficiently achieved. The voltage issues noted in the 
following sequence of events may be viewed in that context. 

Between 10:00 and 11:59 EDT 

• The electrical operator at FirstEnergy’s Eastlake plants reported that system 
voltage problems were experienced “throughout the morning”. 
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• The EPRI team examined voltages around the system for different times of the 
day. At 1 1 :00 AM, a voltage snapshot around the FirstEnergy system showed, as 
expected, that the voltages were higher next to the large power plants and trended 
lower towards the load centers, hi particular, several locations appeared to have 
relatively low voltages (both 345 kV and 138 kV levels) than expected when 
compared to other busses. Some of these anomalies may be attributed to 
calibration of the Capacitive-Coupled Voltage Transformers (CCVT) and 
Potential Transformers (PT). 

• At approximately 11 :54 Cinergy reported to MISO that its Bedford to Columbus 
345 kV line had tripped, resulting in the loss of a 230 kV line segment from 
Bloomington to DeNoye Creek. 

Actual flows (MW, MVAR) measured around the affected area at 12:00 are shown in 
Figure 2. The dark bars represent MW flows. The lighter color bars represent the MVAR 
flows. Negative numbers represent flows from the second location listed to the first. 
Blanks, e.g., AEP-OE, AEP-TE, etc., represent data that were not available to EPRI. The 
purpose of Figure 2 and similar subsequent Figures is to show the import or export of 
both MWs and MVARs into or out of the FE service areas with its neighbors. In Figure 2, 
MVAR was flowing from FE’s Ohio Edison area into Duquesne (DQE). 



Figure 2 - Actual Flows (MW and MVAR) in the Affected Areas at 12:00 PM 

In Figure 2 and subsequent similar Figures, the FirstEnergy areas include OE (formerly 
Ohio Edison), CEI (formerly Cleveland Electric Illuminating), and TE (formerly Toledo 
Edision); DQE is Duquesne, NYPP is NY ISO (formerly New York Power Pool), MECS 
is the former Michigan Electric Coordinating System, DPL is Dayton Power & Light, and 
IMO is Independent Market Operator (Ontario). 
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Between 12:00 and 12:59 EOT 

• 12:00 - Based on data from the historical NERC TagNet for all the electronic tags 
submitted to the NERC Interchange Distribution Calculator (IDC), the total IPP 
generation in SE Ohio from Waterford, Washington, and Hanging Rock was 
scheduled at 2540 MW, ramping up briskly starting at about 8:00 AM. 

• 12:00 - NERC Reliability Coordinators’ conference call activated a lowered 
reference frequency for the Eastern Interconnection for time error correction. 

• 12:04 -FirstEnergy received alarms of low voltages (below 327.8 kV or 0.95 per 
unit) from Harding. 

• 12:05:44 - Conesville Unit 5 (rating 375 MW) tripped. 

• 12:25 - FirstEnergy received low voltage alarms from Inland, Fox, and Harding. 

• Starting at about 12:30, a noticeable and gradual decline in voltages occurred in 
several buses in the FirstEnergy area. For example, by 13:00 the voltage for the 
Star 345 kV bus (a considerably stiff bus with four 345 kV lines and a number of 
138 kV lines) had dropped by over 1% relative to its voltage at 11:00 AM (see 
Figure 3). It can be seen from Figure 3 that the voltage degradation at Star 345 
kV was halted at 1 : 15 PM, corresponding to the increase in VARs from Sammis 

at that time (see Figure 3). That increase follows the System Control Center (SCC) 
request to increase voltage support from all generating plants. During this period, 
there was serious voltage degradation in the Cinergy area and at the same time the 
total generation output at Gibson was high, and the Bedford to Columbus line 
tripped. This voltage degradation and correction at 1 : 1 5 PM are also visible on 
International Transmission Company (ITC)’s Monroe 303 bus, as can be seen in 
Figure 4. 

• 12:46 - A Level 1 TLR was called on the How gate from Ghent to Batesville due 
to heavy flow from Kentucky into Indiana. MISO operators were simultaneously 
monitoring that situation and the Columbus-Bedford situation. 

• 12:5 1:55 - The Dumont 765 kV reactor opened (responding to a need to support 
voltage in the AEP system with that action) 4 . 


4 The significance of this switching cannot be verified without data to ascertain whether this was an isolated 
event or an indication of a wide-area voltage problem. 
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Figure 3 -- Voltage vs. Time at Star 345 kV Bus on August 14, 2003 


'360 kV ; : 

346 kV 

12 1 2 3 .4 EDI 

Figure 4 - Voltage vs. Time at International Transmission Company (ITC)’s 
Monroe 303 Bus (close So FE) on August 14, 2003 
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Figure 5 - Actual Flows (MW and MVAR) in the Affected Areas at 1 :00 PM 
Total FE MVARs = 286 Net Export 
(Note that AEP-OE MVARs = -1; Missing MVARs are AEP-PJM, 
PJM-NYPP, OE-DPL, NYPP-IMO, MECS-IMO, AEP-MECS) 


Between 13:00 and 13:59 EDT 

• 13:00 - The total IPP generation in SE Ohio was scheduled at 2585 MW. The 
actual MW and MVAR flows are shown in Figure 5. 

• 1 3 :00 - Voltage at Star 345 continued to decline. 

• 13:02 - Gavin (AEP, 1300 MW) went on line. 

• 13:10 - The system Control Center (SCC) at FirstEnergy made a request to all FE 
power plants to increase voltage support. At Sammis the operators increased the 
set point and the machine was left on automatic (see Figure 6). 

• 13:14:04 - Greenwood Unit 1 (rating 785 MW) tripped. 

• 13:16 (approx.) - Cinergy and MISO noted that the Wheatland plant had 
synchronized to the system and was generating about 100 MW. 

• 13:23 - Mone 1, 2 (300 MW) went online. 

• 13:30 - In discussing about Wheatland, the MISO operator told the Cinergy 
operator, 5 “We’re aware that when they’re bringing their generation up, it’s 
dropping the voltage. That’s because they’re sucking in more VARs than they’re 
able to put out with their unit. I think that they can only put out 60 VARs 
(probably meaning MVARs), but they’re sucking in close to 300, or not sucking 
in, but the voltage is dropping relative to that amount." 

• 13:31:34-Eastlake Unit 5 (EL5) (rating: 615 MW) tripped, due to MVAR 
runback and subsequent switch from manual to automatic operation to increase 
the VAR output. Immediately after EL5 tripped, as to be expected. Perry 


5 Transcripts of recorded telephone conversations at MISO on August 14, 2003 available at 
http://energycommerce.house.gov/108/Hearings/09042003hearingl062/hearing.htm 


10 




357 


ERrai 

increased both real and reactive power to the Cleveland area. (See Figure 6.) 
Flows on the lines to the Cleveland exhibited increases in MW and MVAR 
loadings. 

• 13:34 - FirstEnergy initiates Automatic Reserve Sharing in response to the loss of 
Eastlake. 

• 13:44 - Mone 3 (150 MW) went on line 

At 13:00, voltages in FirstEnergy’s system continued to decline. At about 13:10, voltages 
started to climb. After Eastlake Unit 5 tripped, the voltage dropped. For example, the Star 
voltage came down to about 98.5% and then stabilized until about 14:00. 



Figure 6 - Reactive Power from Perry Following the Tripping of Eastlake #5 
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Figure 7 - Actual Flows (MW and MVAR) in the Affected Areas at 14:00 
Total FE MVARs = 340 Net Export 


Between 14:00 and 14:59 EDT 6 

• 14:00 - The total IPP generation in SE Ohio was scheduled to peak at 2611 MW 
and the MW and MVAR flows in the affected areas are as shown in Figure 7. 

• 14:02 - Stuart to Atlanta 345 kV line tripped 7 

• 14:48 - Mitchell 1 (800 MW) controlled shutdown 

At 14:10, voltages in FirstEnergy resumed their decline. By about 14:45, the Star voltage 
was stabilized at 335 kV, or about 98.5% of its 11:00 AM value. 

Between 15:00 and 15:59 EDT 

• 1 5:05:41 - Harding-Chamberlain 345 kV tripped for a fault condition. 

• 15:32:03 - Hanna-Juniper 345 kV tripped. Star voltage declined by 18 kV to 322 
kV, or 94.7% of its 1 1 :00 AM magnitude 

• 15:41:33 - Star-South Canton 345 kV tripped, loading about 1272 (SE rating = 
1383 MV A) Star voltage declined by 33 kV to 307 kV, or 90.3% of its 1 1 :00 AM 
magnitude 

• 15:42:53 - Cloverdale-Torrey 138 kV tripped. Loading was 354 MVA (SE Rating 
245 MVA) 

• 15:45:33 - Cloverdale-Canton Central tripped (see Figure 8). Loading on this 
line jumped after Star-South Canton line tripped. Canton Central-Tidd 345 kV 
tripped due to breaker failure at Canton Central. 

• At 15:55 - Star voltage had declined by 37 kV relative to voltage at 11:00 AM 


6 This sequence is not all inclusive as 138 kV lines generally are not included. 

7 From DOE-NERC Time line 
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Figure 8 - MW and MVAR flows on Cloverdale-Canton Central 138 kV line 



Figure 9 - Actual Flows (MW and MVAR) in the Affected Areas at 16:00 
Total FE MVARs =117 Net Export 

At 15:00, voltages in FirstEnergy’s system continued to decline. When Harding to 
Chamberlain tripped, the Star voltage declined by about 1% to 97.3% of nominal. When 
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Hanna- Juniper tripped, the Star voltage declined by another 2,6% to 94.7% of nominal. 
When Canton Central to Tidd tripped, the Star voltage declined by an additional 4.4% to 
90.3% of nominal. Each successive line trip caused the additional voltage drop to become 
more pronounced. This is an indication of severe voltage degradation. Voltage at the 
Cloverdale-Canton Central was 94.9% prior to its trip, according to MISO EMS data. 

From about 2:35 PM to about 3:30 PM, the alarm system of the FirstEnergy system 
appeared not to be performing as designed. As a consequence, the FirstEnergy operators 
may have lacked timely information to assess the developing conditions. During the day, 
other control areas also reported performance problems with some of their hardware and 
software, as indicated in the MISO phone transcripts. 

Between 16:00 and 16:30 EDT 

• 16:06:03 - Sammis-Star 345 kV tripped 

• 16:08:58 - Galion-Ohio Central-Muskingum 345 kV 

• 16:09:06 - East Lima-Fostoria Central 345 kV 

• 16:09 to 16:30 - Numerous line and generator trippings followed resulting in 
wide-spread customer interruptions 

Immediately after Sammis-Star tripped, voltages in the FirstEnergy area declined 
severely. The subsequent disturbances happened so rapidly that human intervention was 
in general not possible. 


Preliminary Incremental MW Load Flow Analysis 

To understand why heavier flows from south to north were observed on August 14, 2003, 
the EPRI team performed a quick and preliminary load flow study using a load flow case 
developed by Commonwealth Associates, Inc. (CAI) 8 The load flow conditions were 
reconstructed using publicly available information and educated assumptions of system 
conditions on August 14, 2003, preceding the series of outages. 

The EPRI study used an incremental approach on top of the CAI base case and was based 
on the following assumptions. 

Study Assumptions 

• The EPRI team performed an incremental analysis of the changes in line flows 
due to the addition of 2610 MW of merchant plant output in SE Ohio. This 
information was not known by CAI and was therefore not modeled in the CAI 
data. Due to limitation of time, we did not model the approximately 230 MW at 
Wheatland, Indiana. (Modeling this in subsequent studies would be important for 
comprehensive understanding of the regional voltage and reactive margins and 
their effect on adjacent areas.) 


s Commonwealth Associates, Inc. (CAI), “White Paper- A Scenario Describing the August 14, 2003 
Blackout", 9/9/2003. 
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• We focused only on the MW flows. The CAI data did not model the reactive 
capabilities available on August 14, and the simulated MVAR flows did not 
match well with the measured MVAR line flows in the FirstEnergy areas. 

• We selected certain lines to monitor in the FE and AEP areas. 

• We used the NERC TagNet data to reconstruct the distribution of the sinks for the 
transactions with the sources at the three merchant plants in SE Ohio. The results 
of that analysis provide the basis for the assumed sources and sinks for this 
incremental load flow case, as shown in the following Table 2. 

Table 2 - Assumed Sources and Sinks for the Incremental Load Flow Study 9 . 



EP'RI Base 

Case 

Change 

Case 

Source(+ve) 
/ Sink(-ve) 

DEWO Net Export (MW) 

- 

610 

610 

Waterford + DELO (MW) 

- 

2,000 

2,000 

IMO Net Export (MW) 

(1,600) 

(2,200) 

(600) 

CIN Net Export (MW) 

287 

237 

(50) 

MECS Net Export (MW) 

422 

172 

(250) 

CE Net Export (MW) 

1,445 

1,365 

(80) 

FE Net Export (MW) 

(3,613) 

(3,763) 

(150) 

Assumed AEP Purchase 

- 

1,201 

(1,201) 

System Losses (MW) 

14,667 

14,946 

(279) 

Total 



0 


The results of the incremental load flow study are highly dependent on the assumptions in 
Table 2, especially where the sinks were located and where the re-dispatched generators 
in the AEP system were located, in order to absorb the incremental generation from the 
IPPs. As is done routinely in such wide-area transfer studies, the area swing generator in 
each importing control area will be automatically used by the load flow program for 
reducing generation output to achieve a desired interchange schedule. In the AEP system, 
however, the area swing generator was Spom, which was located close to those merchant 
plants. We allowed Spom to be used as the swing bus, even though in reality, the actual 
generators re-dispatched to absorb the purchases would likely be the most expensive units 
on line at that time. As we did not have such data about the actual generation dispatches 
in AEP, we used Spom as the swing bus, even though that might underestimate the 
incremental flows across the AEP grid. 


Also, because the Eastern Interconnection at heavy load conditions is highly congested, 
load flows often run into convergence problems when large changes are made to the 
desired interchange schedules. As a result, the incremental results were obtained by 
breaking up the changes into six cases. Along the way, it was noted that Spom was 
generating a significant negative MW value (-1357 MW) and reached its MVAR limit, 
and the load flow could not converge in the study. To make it converge, we backed off 
manually by 700 MW the Kammer generation, which is close to the Muskingum River 
station near the Waterford and the Washington plants. That is to say, we did not back off 
AEP generation in the western part of its system to influence the incremental flows. 


' NERC TagNet data are generally reported as inter-control area schedules. They exclude schedules from 
other sources inside a control area with sinks designated to be also inside that control area. 
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One assumption that would influence the results was that Ontario (IMO) was importing 
600 MW more than the amount in the base case. This is an assumption based on the tact 
that IMO was importing more power on August 14 than was typical, and that it was 
increasing its import continuously between 12 PM and 3 PM that day. Therefore, it is 
conceivable that some inadvertent interchanges during that period might have found IMO 
to be their sink. Another possibility is that some control areas might have wheeled more 
power to IMO at that time. Again, with more actual dispatch and interchange schedule 
data available from all control areas, the study can be refined. 

Incremental Loud Flow Results 

The results of the incremental load flow study are summarized m Figure 10 below. 


Increases in MVA Loading due to 2610 MW of Output from IPPs in SE Ohio 
(Maximum Schedule on August 14, 2003) 

Based on Preliminary Load Flow Simulations 


Figure 10 - Estimated Increases in the MVA Loadings of Selected Transmission 
Lines in FirstEnergy and AEP Due to 2610 MW of Output from IPPs in 
SE Ohio (Maximum Schedule) on August 14, 2003 

(The simulated line flows on these lines carried very low MVARs. Thus the MW 
loadings on these lines were approximately equal to the MVA loadings.) 

From the assumptions about the sinks of the scheduled generation, about 1480 MW ol the 
IPP output was absorbed by AEP or went towards increased system losses. Thus AEP 
exported about 1 130 MW. Figure 10 shows that of this 1130 MW, about 540 MW (170 
plus 370) flows on two transmission paths into central Ohio from Muskingum River. 
Also, about 140 MW flows from Ashtabula to Erie West, which is in the direction to the 
Northeast. 
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Significantly, numerous assumptions and inherent modeling inaccuracies were 
necessarily adopted for this preliminary power flow analysis. There is a need for 
' additional detailed data to accurately assess the system conditions and sequence of events, 
including specific generator dispatches, reactive outputs and capabilities, status of other 
reactive devices and control settings, transformer tap settings, phase shifter settings, 
capacitor banks, SVCs, HVDC converters, scheduled and actual transactions, etc. Since 
reactive conditions and power transfers appear significant in the analysis, a concerted 
effort by all affected entities is required to benchmark refined models against actual 
system measured data to validate the study results. 

The normal MVA rating of the Muskingum River to Ohio Central line is 938 MVA in the 
CAI data, which was based on the NERC MMWG (Multiregional Modeling Working 
Group) case for the summer of 2003. Therefore, the 170 MW of incremental flow on the 
Muskingum River line amounts to about an 18% increase in the loading of the line. In the 
CAI base case, that line was already loaded to about 94% of normal rating. With the 
change case, the line loading went to 1 12% of normal rating. Without more data and 
more computer studies, the overload on that line or other lines in the AEP system during 
August 14, 2003 cannot be confirmed. However, this is a hypothesis that deserves further 
investigation. 

In summary, the incremental load flow study shows that if the actual generation outputs 
of the merchant plants in SE Ohio on August 14, 2003 corresponded to the schedules 
obtained from the NERC TagNet data, they may have increased the flow through Central 
Ohio towards Northern Ohio and Michigan. Part of the flow, though to a lesser amount, 
would have gone along Lake Erie towards the Northeast. If Ontario was importing more 
power at that time, part of that generation may have ended in Ontario, mostly via 
Michigan and partly via New York. 

In short, this incremental study indicates that this hypothesis is an area that deserves 
further data collection and analysis. 

Was August 14, 2003 Typical and Were the Series of 
Disturbances Related? 

In an attempt to answer the question whether August 14, 2003 was in someway special or 
different, we looked at data available from TagNet for all hours and all days from August 
1 to August 14. We computed the net exports for most of the control areas in the 
northeast. We plotted the 24-hour weekday profiles of their net exports and computed the 
average weekday profile from Aug 1 to Aug 13. We plotted each of the weekdays and the 
average profile. From these graphs, we found that August 14 was distinct from the early 
August weekdays for AEP, MECS (Michigan Electric Coordinating System), FE and 
MO (Independent Market Operator of Ontario). For some of them, August 13 was 
similar to August 14. To illustrate, the graphs for AEP and 1MO are shown in Figures 1 1 
and 12. 
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A£P Weekday Nat Export (Aug 1-13) 



Figure 1 1 - AEP Weekday Net Export Schedules (Aug 1-13, 2003) 


IMO Weekday Net Export (Aug 1-13) 



Figure 12 - Ontario Weekday Net Export Schedules (Aug 1-13, 2003) 

The data obtained from TagNet about the schedules of the SE Ohio merchant plants seem 
to be consistent with the reduced net exports out of AEP’s own generation on August 12, 
13 and 14. The higher imports by Ontario on these same three days seem to be consistent 
with the increasing hot temperatures on those days. Both sets of curves appear to lend 
some possibility to the incremental load flow studies associated with the operation of the 
IPPs in SE Ohio. 

While August 13 and 14 were similar, insights as to why the disturbances happened on 
August 14 and not on August 13 can perhaps be found in the voltage behavior on those 
two days, which may be different. To help shed light on the voltage behavior, we 


18 


365 


ERiai 

obtained hourly voltage profiles from August 8 to August 14, 2003 for the Star 345 kV 
bus m the FirstEnergy area. Integrated data for August 14, hour 15 was unreliable due to 
disturbances, so subsequent data are also not included. This is plotted in Figure 13 and 14. 


STAR-345 kV BUS 
Aug 8-14, 2003 


Fri Sat Sun Mon Tue Wed Thur 

8 9 10 11 12 13 14 



Figure 13 - Star 345 kV Bus Voltages (Aug 8-14, 2003) 
Lowest normal operating range = 327.8 kV (95% of 345kV) 


Figure 1 3 clearly shows a trend in the system voltage that was not only noticeable voltage 
degradation during each day, but also a noticeable worsening of the voltage problem in 
the three successive days preceding August 14, 2003. Additional data for other days 
preceding August 8 would be useful to validate this observation that August 14 was 
atypical. Based on this data alone, however, it supports the need to investigate the causes 
of the successive voltage degradation in the tliree days preceding the outage. 

Figure 14 narrows the analysis into the hours preceding the major disturbances at about 4 
PM on August 14, 2003 and compares the voltages during those hours in the preceding 
weekdays. The curves show that August 12, 13, and 14 were clustered together as 
compared to August 8 and 11. Again, the successive worsening of the voltage profiles in 
those three days is quite apparent. It can further be observed that on August 12, the 
voltage reversed its downward trend and began to increase at about 13:00 EDT. On 
August 13, the voltage continued to drop for another hour before increasing at 14:00 EDT. 
On August 14, the voltage dropped at a faster and faster rale starting at 12:00 and did not 
show any sign of reversing its downward trend, in contrast to prior days. Higher 
afternoon temperatures, greater and increasing power requirements in the Eastern 
Interconnection and the status of generating units could have contributed to lowering the 
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overall reactive reserve margin on August 14. Perhaps the loss of three generators on 
August 14 contributed to lowering the overall reactive reserve margin on that day. 


STAR-345 kV BUS 
Aug 8, 11, 12, 13, and 14, 2003 



Figure 14 - Star 345 kV Bus Weekday Voltages (Aug 8, 11, 12, 13 and 14, 2003) 

Another factor that was documented by the MISO operator phone transcripts and 
described in the timeline at 13:30 was the possibility that merchant plants are not all 
designed to provide sufficient reactive power output. The Wheatland plant had four 125 
MW simple cycle gas turbines and was rated at 508 MW 10 . Thus when 2 units were 
online shortly after 14:00, its maximum rating would be 254 MVA. It was scheduled to 
output 236 MW, and apparently the MISO operator had reason to believe that it was 
capable of only 60 MVARs. In terms of capacity factor, it was operating at 236/254 = 
93%. If indeed it was generating 60 MVARs, its power factor at the time would be about 
97%. Either calculation would support the MISO operator’s concern about voltage 
support. Similar calculations place the Hanging Rock plant (1240 MW capacity) 11 
running at 1200/1240 = 97% capacity factor from 13:00 on. Likewise, the Washington 12 
and Waterford 13 plants were scheduled to output at 15:00 EDT 61 1 MW and 800 MW, 
respectively. Compared to their plant ratings of 620 and 850 MW, they were running at 
98% and 94% capacity factors, respectively. Furthermore, in order to absorb 2600 MW of 
merchant plant output from SE Ohio, even with the incremental losses estimated to be 
about 280 MW, about 2320 MW of other power plants would have to be reduced in 


5 0 http :// w wvv.ail eghenyen ergysuppl y . coin/' genera ting'd efault.asp 
i! htl|x//www. duke-energy. coni/businesses/p lan ts/own/us/midwest/hangingroek. asp 
l ~ http://www.duke-energy.eom/businesses/plants/ow:i/us/nudwest/washingtoo.asp 
!} hup://\vww.pscg.com/mcdia_center/pressreleases/articles/press_20O3-O8-l l.html 
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output or shut down. If part of this was achieved by shutting down some units, then some 
reactive support would be lost from the Eastern Interconnection. While there is no data 
available to the EPRI team on how much reactive power was put out by other power 
plants in that area to compensate for the lack of reactive power output from the merchant 
plants, it certainly raised the question whether it was sufficient to maintain voltages for 
the immediate area and not create a burden on adjacent areas. Thus, while no definitive 
conclusion can be made at this time, this is certainly an area deserving further 
investigation. 

Changing Patterns of Power Flow and Frequency Behavior 

When an interconnected power grid is heavily congested, as is the case with the Eastern 
Interconnection, incremental flows are significant contributing factors to the total loading 
of critical transmission bottlenecks. This preliminary study illustrates the importance of 
simulating different wholesale power transfer patterns in the Eastern Interconnection. It is 
well known among transmission planners in the Eastern Interconnection that the 
wholesale power transfer patterns have changed to a significant degree since restructuring 
and the FERC Order 888 in 1998. New bottlenecks have emerged as transfer patterns 
change. New power plants, their real and reactive ratings, and their locations also have a 
major effect on the wholesale power transfer patterns. 

With increasing number of wholesale power transactions starting and ending at the same 
hours due to common business practices, the Eastern Interconnection has experienced 
larger fluctuations of system frequency, especially during morning load pickups, 
afternoon drop offs and at about 10 PM or 1 1 PM at night when many transactions end at 
the same time. Unanticipated frequency excursions that occur regularly due to inadvertent 
interchanges may degrade the region’s ability as a whole to balance generation and load 
in response to sudden disturbances. This is another systemic problem that deserves 
further consideration. 

Studies done under the new wholesale market environment should be reviewed regularly 
to determine if their conclusions are still valid. The study approach and the study tools 
should also be re-examined, in light of the need to consider a large number of 
simultaneous transfer patterns. For example, the approach taken by the NERC Pre-Season 
Assessment Study Team (PSAST) supported by EPRI’s Transmission Reliability 
Initiative for assessing the reliability of the Eastern Interconnection for the summer of 
2002 14 made use of a combination of a simultaneous transfer capacity program, TRACE, 
and a new software called the Community Activity Room (CAR). The CAR is also being 
tested by EPRI and TVA as an online congestion monitoring tool. 

Rapid Online Assessment of Transmission Bottlenecks 

In addition to the greater need to update planning studies, the rapidly changing transfer 
patterns within a day would seem to call for more rapid online studies of transmission 
bottlenecks over the entire Eastern Interconnection. In such online studies, the actual line 


1 ' ) “Summary Report: Summer 2002 Eastern Interconnection Pre-Season Study and New Tools for 
Community Activity Room”, EPRI Report, July, 2002. 
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outage status of all critical transmission lines must be modeled as they take place. 
Topology Estimator, which is a super State Estimator program capable of detecting the 
correct network topology of a wide area such as an Interconnection, is a technology that 
has recently become available. It appears that a Virtual RTO 15 concept proposed for 
integrating the communication and computing infrastructures of the physical RTOs or 
ISOs of an Interconnection would be a potential platform to implement an 
Interconnection- wide Topology and State Estimator. 

It is also vital that all the operators in an interconnection see the same bird’s-eye view of 
the entire interconnection when it comes to managing congestion. Using geographically- 
based transmission flow displays driven by the same real-time data will help. Also, the 
Community Activity Room offers a radar-screen-like visual display of congestion in any 
major bottlenecks in the entire Interconnection, either in 3D or 2D. With such display 
tools, all the RTOs and grid operators in the same interconnection can coordinate 
effectively to relieve congestion that requires wide-area coordinated measures. When 
long distance and wide-area wholesale power transactions are significantly impacting 
congestion, local measures are ineffective. Such wide-area display systems will enable 
grid operators to handle situations before they develop into emergencies. Upgrading data 
collection capability and analytic tools with systems that are now available or under 
development is likely a necessary requirement to assuring grid control and stability under 
the current pervasive conditions. 

Reactive Reserve Margins 

Another area deserving further investigation is the reactive reserve margins in all parts of 
the Eastern Interconnection. Little is known about the actual amount of dynamic and 
static reactive capacities online at any given time in various parts of the grid. Since 
restructuring and open access, power plants that are not directly controlled by grid 
operators do not typically have enough financial incentives or contractual obligation to 
generate reactive power. Without sufficient reactive sources at the generating end of the 
high voltage grid, voltage support in a wide area cannot be achieved effectively. In 
addition to not supporting grid voltage, generators that operate at close to unity power 
factors in some situations may consume reactive power from the grid, either directly or 
through the reactive line losses from the increased line flows from their plant outputs. 

Therefore, to ensure grid reliability, voltage support is now an important and critical 
concern of the grid operators. It is clear that grid operators do not have an accurate real- 
time picture of the reactive reserve margins in their own service areas, due to the lack of 
data about the merchant plants. 

There are many ways deficiencies in reactive power can develop in a system that 
seemingly had adequate resources. A power system may be able to progress through one 
day’s heavy load periods with little or no low voltage problems. This same system, 
however, can experience low voltages if it had major contingencies of generators or 
transmission lines, heavy power transfers through its systems, or customers within the 


15 Stephen Lee, “Executive Summary of Virtual Regional Transmission Organizations And the Standard 
Market Design”, EPRI Report, Product ID 1007680, February 5, 2003. 
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system operate their equipment differently. The greater the level of power transferred 
through the system, the greater the reactive power losses and voltage drops will be. 
Conversely, reactive power losses are directly related to system voltage levels. The lower 
the voltages, the heavier the reactive losses are. If the active power transfer on the system 
is high, even small increases in MW transfer could lead to large increases in MVAR 
usage. Figure 15 illustrates this concept. 

In this textbook example, two 200 mile, 345 kV lines are carrying equal load (about 470 
MW, and 30 MVAR). Their initial operating point is labeled in the Figure 15 as “Two 
Lines In”. When one line trips, the remaining line would carry the total load (assuming 
the radial configuration shown). The new operating point is labeled as “I Line In”. As 
can be seen from this figure, the MWs on the remaining line would double, but the 
MVAR would dramatically increase to about 1 100 MVAR. This sudden increase in 
reactive losses potentially escalates reactive demand on the system, reduces reactive 
reserves, and increases the risk of a gradual voltage collapse. Furthermore, the increased 
reactive losses results in a higher line current for the same amount of MW flow, thus 
loading this textbook line beyond its thermal rating. The line under such conditions 
would sag, hence increasing the chances of contacting trees. The additional MVAR 
would have to come from the system, which would increase the flows on other lines, 
further increasing reactive losses, assuming that the system has enough margin to support 
the increased VAR demand. This phenomenon could lead to increased depletion of 
reactive reserves on the system, potential voltage degradation under increasing system 
stress, and possibly resulting in voltage collapse and line and unit trippings. 



Figure 1 5 - How Losing A Parallel Line Would Compound 
the Overload Problem When MVAR Flow is Heavy 
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While Figure 15 is an example from a textbook, Figure 16 is for the loading of a 138 kV 
line in the FirstEnergy System. Notice the similarities in the phenomena. 



Figure 16 - MVAR load increase in the West Akron to Ira line 


Figure 17 shows the MVAR flows on four transmission lines at the West Akron substation in 
FirstEnergy following the tripping of adjacent lines. Not all the lines from the West Akron 
substation are shown for clarity. The graph shows that as one line trips, adjacent lines pick up 
both MW and MVAR flows, and MVAR flows increase more than MW flows. 



Figure 17 - MVAR Flows in Adjacent Lines Following Line Trips 
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Wide-Area Voltage Problems 

Voltage problems can spread to adjacent areas because an area which is deficient in 
reactive power will draw on neighboring systems for reactive support, thus dragging 
down the voltages of the adjacent systems. 

A good analogy to use for understanding reactive power and voltage support is that of a 
tent made of heavy canvas. The tent covers a wide-area grid. The level of the tent at 
various parts of its canvas represents the voltage level. In order to prevent the sagging of 
the canvas from causing it to touch the ground, one can visualize a set of powerful water 
jets spaced out underneath the canvas. The forceful water jets support the tent at the 
points where they impact the canvas. The weight of the tent is then delicately balanced by 
the water jets. If one then places heavy objects on the canvas to represent reactive loads 
on the grid, in order to carry these additional weights, one must increase the outputs of 
the water jets at the most effective locations underneath the tent. Without these additional 
forces from underneath to support the tent, the additional weight will cause the canvas to 
sag and sag and eventually the whole tent will collapse. 

Another insight from this analogy is the reactive power flows, or MVARs. Reactive 
power flows from high voltage to low voltage. When a water jet pushes that part of the 
tent above the adjacent parts, reactive power will flow from where the tent is higher to 
where it is lower. In doing so, it helps support the adjacent parts. Therefore, in a situation 
where the voltage in an area is low, if the reactive power flows from that area to the 
adjacent areas, this would be an indication that the canvas in the adjacent areas is even 
lower. This was shown in Figures 5, 7, and 9 for 1 PM, 2 PM, and 4 PM, with 
FirstEnergy being a net exporter of reactive power to its neighbors in the amounts of 286, 
340, and 1 17 MVARs, respectively. A natural question to ask is “how much of the 
sagging in the FirstEnergy area is caused by the lack of water pressure in the water jets in 
that area or the dragging down by the adjacent areas because of the lack of water jet 
pressures in those other areas?” 

Areas for Further Investigation 

In summary, this white paper has presented data and some preliminary study results to 
suggest that the events of August 14 were not independent random coincidences. Two 
potential contributing factors to the power outages were long distance wholesale power 
transfers and wide-area voltage behavior. They combined to create the operating 
environment in which even small disturbances, when added together, contributed to the 
widespread disturbances. To explore this hypothesis further, additional data for the real 
(MW) and reactive (MVAR) capacities and the real and reactive loads in all parts of the 
Eastern Interconnection during August 14, 2003 are required so that an overall 
interconnection-wide load flow and voltage study can be done. 

In the course of this study, EPRI did not have sufficient time or resources to draw 
definitive conclusions or make final recommendations to transmission providers, grid 
operators, and ISOs/RTOs directly impacted by the outage or to the industry at large. 
However, based on the results of analysis and the knowledge and experience of the EPRI 
team, areas for further investigation have emerged. The EPRI team has provided two 
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lists of areas for further investigation. The first list is specifically for transmission 
providers, grid operators, and ISOs/RTOs directly impacted by the outage. The second 
list is for the broader industry at large. These are listed below, not necessarily according 
to any priority order. 

Areas for further investigation for transmission providers, grid operators, and ISOs/RTOs 
directly impacted by the outage: 

1 . Investigate the need for reactive support. 

2. Investigate any unexplained transmission line tripping. 

3. Review emergency ratings of transmission lines and safe operating limits, including 
voltage and stability limits and consider monitoring of real-time dynamic line limits. 

4. Review line relay malfunction and settings under different flow patterns. 

5. Investigate alarm system, data communication, Internet access, and Energy 
Management System hardware and software performance on August 14, 2003 so that 
improvements can be made to prevent malfunctions from affecting the response of 
grid operators to future disturbances. 

6. Review system operation procedures, emergency action response instructions, real- 
time information sharing and communication protocols among operators, MISO 
operators, and adjacent systems’ operators. 

7. Review under frequency and under voltage load shedding and system separation 
schemes. 

8. In parallel and in coordination with the studies done by NERC or the U.S./Canada 
Power Outage Task Force, model the individual entity system response in the Eastern 
Interconnection, in particular paying close attention to modeling wide-area and long 
distance wholesale power transfer patterns including the operation of independent 
merchant power plants. 

9. In parallel and in coordination with the studies done by NERC or the U.S./Canada 
Power Outage Task Force, simulate similar conditions and determine required 
responses to avoid future disruptions, including a comprehensive contingency 
analysis and probabilistic reliability assessment. 

Industry Recommendations: 

1 . Collect and review additional data from all sources in order to study the wide-area 
and evolving situations on August 14, 2003. In particular, data about merchant plant 
scheduling and operation, reactive capacities and reactive reserve margins in the 
entire Eastern Interconnection, are extremely important to performing a complete 
analysis of the contributing factors to the power outage. The problem does not appear 
to be an isolated incident of unlikely random failures. Potential systemic problems 
associated with communication and coordination among the reliability entities, 
generators, marketers and transmission providers should be assessed. 

2. Review the current methods for scheduling power across the Eastern Interconnection 
to ensure that all simultaneous transfer limits are recognized, and model system 
response in the Eastern Interconnection, in particular paying close attention to 
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modeling wide-area and long distance wholesale power transfer patterns including the 
operation of independent merchant power plants. 

3. Review the adequacy, applicability, and utilization of wide area electronic exchange 
of operating data to address potential improvements. 

4. Investigate the need for reactive support in specific areas or in the Eastern 
interconnection as a whole using computer studies. 

5. Simulate similar conditions and determine required responses to avoid ftiture 
disruptions, including a comprehensive contingency analysis and probabilistic 
reliability assessment. 

6. Review under frequency and under voltage load shedding and system separation 
schemes. 

7. Investigate alarm system, data communication, IDC and Energy Management System 
hardware and software performances on August 1 4, 2003 in all the control areas 
affected by the disturbances so that improvements can be made to prevent any such 
malfunctions from affecting the response of grid operators to future disturbances. 

8. Investigate all transmission line trippings to determine potential improvements. 

9. Review emergency ratings of transmission lines, and safe operating limits, including 
voltage and stability limits and consider monitoring of real-time dynamic line limits. 

10. Review line relay malfunction and settings under different flow patterns. 

1 1 . Review system operation procedures, emergency action response instructions, real- 
time information sharing and communication protocols among grid operators and 
between ISOs or RTOs. 

12. Review the relevance and adequacy of NERC’s planning and operating. guidelines. 

13. Review protection schemes of power plants (including merchant plants) and their 
coordination with the protection of the interconnected grid. 

14. Review the adequacy of transmission impact studies conducted by an RTO or a 
transmission provider to connect a new power plant to the grid, especially in view of 
the needs to include complex dynamics and voltage stability that are sensitive to 
changing transfer patterns, and review the adequacy of NERC standards related to 
reactive capability, voltage support, and other ancillary capabilities by all power 
plants, including merchant plants. 

15. Review data archiving requirements and accessibility. A major difficulty for post- 
disturbance analyses is the lack of availability of adequate historical data related to 
critical infrastructure elements. An industry standard for the required data (e.g., 
voltages, flows, generation and loads, reactive support device status, etc.), common 
formats and archiving requirements for different classes of data appear to be needed. 
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MICHIGAN PUBLIC SERVICE COMMISSION 
REPORT 

BLACKOUT OF AUGUST 14, 2003 


EXECUTIVE SUMMARY 


The August 14, 2003 blackout was a wake-up call concerning reliability of our nation’s electric 
grid. What started out as a typical warm summer day, which looked largely uneventful to most 
electric reliability coordinators (and others) in the northeastern region of the country took a sharp 
u-tum at approximately 4.T0 p,m., when, in a matter of seconds, 50 million North Americans 
found themselves without power. North America’s largest ever outage stretched from 
southeastern Michigan through Ontario and northern Ohio, all the way east to New York City. 
Through this event, North Americans were abruptly reminded how vital electricity is in our 
everyday lives and how tightly interconnected and vulnerable this country’s electric grid has 
become. 

The Public Service Commission did not attempt to determine the root cause of the blackout. 
However, our investigation did not reveal any evidence that Michigan utilities or transmission 
operators were responsible for the blackout. All of the transmission line and power plant outages 
that occurred in the two and one-half hours preceding the power surges that precipitated the 
blackout involved the facilities of FirstEnergy and American Electric Power in Ohio. These 
events led to two large power surges as power from southern Ohio attempted to reach load in 
northern Ohio. The first surge was from southern Ohio, west to Indiana, north to western 
Michigan, east to the Detroit area, and south to northern Ohio. This surge resulted in the opening 
of interconnections in central Michigan between the western part of the State and the Detroit 
area. These interconnection trips occurred as designed to prevent damage to equipment from the 
power surge. The second power surge involved a giant loop from southern Ohio to Pennsylvania 
to New York to Ontario to Michigan to northern Ohio. This surge resulted in the blackout 
around what is generally referred to as the Lake Erie Loop. 

Michigan utilities and transmission companies were not notified of the problems being 
experienced by FirstEnergy and American Electric Power and received no advance warning of 
the potential blackout. The first indication in Michigan of an impending emergency occurred at 
4:09:27 p.m. when an interconnection in central Michigan exceeded its emergency rating as a 
result of the first power surge coursing through the State. A minute later the power outages 
began and by 4: 1 5 p.m., the blackout was complete. A total of 2.3 million customers of The 
Detroit Edison Company, Consumers Energy Company, and the Lansing Board of Water and 
Light were left without power. 


Our investigation leads to the conclusion that electric reliability has been seriously compromised 
by the fragmented and ineffective regulation of the electric transmission system. The Midwest 
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market is coordinated through two regional transmission organizations (RTO), rather than one. 
Moreover, the two RTOs are voluntary organizations that do not cover contiguous territories but 
rather are intermixed in a checkerboard fashion. This “Swiss-cheese” approach to coordination 
prevents any one entity from comprehending the overall situation. The situation is exacerbated 
by a lack of enforceable reliability standards. The North American Electric Reliability Council 
(NERC) is responsible for the development of procedures for reliability coordinators, but lacks 
the authority to enforce those standards. A NERC investigation of compliance in 2002 found 
that there were 444 violations of operating measures totaling $9 million in “simulated sanctions”. 
In addition, the Federal Energy Regulatory Commission (FERC), the agency responsible for 
economic regulation of transmission, indicates that it lacks authority to develop or enforce 
reliability standards. 

In our opinion, the simulated enforcement of reliability standards is inadequate to protect 
Michigan or the nation’s citizens. We recommend that the FERC be authorized to require 
membership in a single transmission organization for each region and have the jurisdiction to 
mandate the development of reliability standards and enforce those standards with real rather 
than simulated sanctions. 

With regard to recovery from the blackout, our investigation reveals that Detroit Edison, 
Consumers Energy, and the Lansing Board of Water and Light performed appropriately. 
However, we conclude that there were two factors that caused restoration in Michigan to lag 
behind other States. First, Detroit Edison’s computerized dispatch system was inoperable due to 
the blackout, which required additional time and effort for the restoration. We recommend that 
the utility conduct a study of potential modifications to the system and report to the Commission 
on the results. Second, the failure of rupture disks at four of the Detroit Edison generating plants 
slowed the pace of restoration. Since rupture disks are a feature designed to protect against more 
serious damage to the units, this does not necessarily indicate a problem. However, we are 
recommending that Detroit Edison analyze the operation of the rupture disks on its units, 
including a comparison with the operation of disks on other utility systems affected by the 
blackout, to determine whether any changes are warranted. 

Finally, with respect to emergency planning and response efforts, we conclude that the 
operations conducted through the State Emergency Operations Center (SEOC) were effective in 
implementing the emergency response plans. However, we note two important improvements 
that can be made to better prepare for future contingencies. First, the Commission Staff 
members who participated at the SEOC were volunteers. We conclude that Staff for the SEOC 
should be assigned in advance and receive training in the operations required to implement the 
emergency plans. Second, we note that the existing emergency electrical procedures were 
adopted in 1979 and have not been reviewed since. Although those particular procedures were 
not needed in this instance, we conclude that it is time for them to be reviewed and, if necessary, 
updated. 
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INTRODUCTION 


The August 14, 2003 electricity blackout stamped an indelible impression on the minds of North 
Americans. Stretching from as far west as Detroit, the blackout covered much of Ontario, 
Canada, northern Ohio and extended all the way east to New York City. Almost as quickly as 
the event struck an avalanche of worldwide media coverage reported that the largest electric 
blackout in North American history had, in a matter of minutes, suddenly plunged 50 million 
North Americans into darkness and forced thousands of businesses to abruptly close operations. 
In its wake a renewed appreciation of the importance of electricity in all aspects of our everyday 
lives was stirred along with a rekindled understanding of just how intricately interwoven, 
interdependent, and vulnerable our electrical system has become. A search for answers as to 
what happened on August 14, and, more importantly, what can be done to strengthen the 
reliability of our electric system to prevent such an event from recurring in the future was 
immediately set in motion with a sense of keen urgency. With over six million residents out of 
power for up to two days and hundreds of businesses shut down, some for several days, 

Michigan elected to commence an investigation to examine the blackout from our vantage point. 

The Michigan Public Service Commission is one of a number of entities commencing an 
investigation to examine what went wrong on August 14 and, more importantly, to determine 
how such an event can be prevented from recurring in the future. The recommendations in this 
report are based upon an analysis of those events in and around the State that resulted in the 
blackout. The report is organized into five parts. Part I - Facts and Overview - presents a 
summary of the electric power system, the relevant actors in that system, and the timeline of 
events on and after August 14. Part II - Electric Transmission - analyzes the operation of the 
transmission system, including operators and regional oversight organizations. Part III - Utility 
Operations - analyzes the operation of utility distribution systems and the efforts of utility 
personnel to recover from the blackout. Part IV - Emergency Planning and Response - 
discusses the response of the Michigan Public Service Commission and other State government 
agencies in the recovery operations and discusses the interdependencies among the various 
infrastructures that were affected by the blackout. Finally, Part V - Conclusions - presents a 
summary of the recommendations put forth in the report. 

This report was prepared by the Michigan Public Service Commission and its Staff. The 
Commission wishes to thank Gary Kitts, Lisa Molner, Jeff Pillon, and Paul Proudfoot, who were 
the primary investigators, along with Robin Barfoot, Bill Bokram, Tim Boyd, Angela Butcher, 
Bill Celio, Mike Fielek, Mick Hiser, John King, Steve Paytash, and Linda Stevens. Information 
was obtained from various entities involved in the power production and delivery system, 
including utilities, transmission companies, and regional transmission organizations. 

Information was also provided by other State government agencies - the Commission wishes to 
acknowledge the assistance of Bob Tarrent and Celeste Bennett of the Department of 
Agriculture; James Cleland, Water Division, Department of Environmental Quality; Captain Dan 
Smith, Michigan State Police; Eileen Phifer, Department of Transportation; Dan Lohrman, 
Department of Information Technology; and Colonel Mike McDaniel, Homeland Security 
Advisor to the Governor. 
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The events of August 14 are reminiscent of a scene from the 1950 movie The Day the Earth 
Stood Still . Professor Bamhard was talking to his secretary Hilda about the worldwide electric 
blackout that had been caused by alien visitors to Earth. He asked her, “Does this make you feel 
insecure?” and she said “Yes.” His response was “I am glad,” because he hoped that the 
blackout would bring about change. The Commission does not wish to suggest that anyone 
should be glad about the events of August 14, but we believe that the recommendations in this 
report will, if implemented, bring about the changes necessary to ensure a more reliable electric 
system for all. 
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PART 1 


FACTS AND OVERVIEW 


Section 1.1: Overview of the Electric Power System 

We have come to expect it - flip the switch and the light comes on. It is something that almost 
every child, from the age of two, leams as the natural order of things. However, this everyday 
commonplace would have seemed like magic to those predating the work of Thomas Alva 
Edison. Indeed, a large, complex system of organizations and infrastructure is required for the 
lights to stay on. 

Generally, the electric power system is divided into three components: generation, transmission, 
and distribution. 

Generation, or the act of producing electricity, is, for the most part, carried out at large power 
plants, which convert another energy source to electricity. Although the specific details vary, in 
general, fossil fuel plants burn coal, oil, or natural gas, use the resulting heat to convert water to 
steam, and then run the steam (or the heated air from combustion) through turbines to create 
electricity. Other fuels, such as landfill gas or municipal solid waste, can be substituted in 
essentially the same process. The generating process in nuclear plants is similar, except that the 
heat is derived from the fission decay of radioactive elements. 

Electricity can also be generated by non-thermal means. Hydroelectric plants generate electricity 
by directing falling water through turbines. Electricity can also be produced using wind-driven 
impellers to turn the generating unit. In addition, sunlight can be used to generate electricity in 
photovoltaic cells. Based on data (through June 2003) published by the Energy Information 
Administration, the current mix of electric generation in Michigan and nationally is as follows: 


Fuel Type 


Michigan 

United States 

Coal 

62.6 % 

51.0% 

Nuclear 

23.3 % 

20.1 % 

Natural Gas 

10.2% 

17.2% 

Renewable Power 1 

2.5 % 

2.1 % 

Petroleum 

0.8 % 

2.8 % 

Hydro 

0.6 % 

6.8 % 


The transmission function involves the large-scale movement of power from generating units to 
the distribution networks, which then deliver that power to the customer. Transmission is 
distinguished from distribution in that transmission lines are larger, operate at significantly 
higher voltages, and individually deliver much larger amounts of power. Transmission can be 


1 The Energy information Administration definition includes: wood, black liquor, municipal solid waste, landfill 
gas, sludge waste, tires, agricultural byproducts, biomass, geothermal, solar thermal, photovoltaic, and wind. 
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fairly described as the bulk transport of power primarily at wholesale, while distribution is the 
delivery to the customer of smaller amounts of power at retail. 

In the past (and still to a great extent today), all three functions were performed by a single 
entity. Investor-owned utilities are usually large private companies that generate their own 
power and serve customers in designated franchised service territories. In Michigan, there are 
two large investor-owned utilities (Detroit Edison and Consumers Energy) that between them 
serve more than 80% of the State and seven smaller ones. 2 Cooperative utilities are member- 
owned companies that normally serve rural areas. There are 12 cooperative utilities in Michigan. 
Municipal utilities are publicly-owned organizations that serve the local municipality and may 
serve some adjacent areas. There are 41 municipal utilities in Michigan. In some cases, 
cooperative and municipal utilities own generation, but, in others, they enter into joint generation 
agreements. Both approaches are used in Michigan. 

In recent years, three new types of entities have begun to enter the electric utility market. 
Independent power producers are private companies, not associated with the local utility, that 
build and operate electric generation plants and sell the power output into the market. In 2002, 
independent power producers built four new generating plants in Michigan. 

Alternative electric suppliers are private companies that sell power at retail in competition with 
the local electric utilities. Alternative electric suppliers do not own distribution lines or deliver 
the power - they rely on the local utility to do that for a fee. To date, the Commission has 
licensed 26 alternate electric suppliers to operate in Michigan. 

Independent transmission companies are private companies that own and operate one or more 
transmission systems. The two largest electric utilities in Michigan have sold their transmission 
assets to independent companies. Hence, the utilities now own the generation plants and 
distribution networks, but independent companies own the transmission that connects the two. 

There are other organizations that are not directly involved in operations, but can have a 
significant impact on reliability. Regional transmission organizations (RTO) are composed of 
transmission companies (both independent and utility) within a given region. RTOs are 
responsible for coordinating access to and use of the transmission network. RTOs are subject to 
approval by the Federal Energy Regulatory Commission, which has jurisdiction over wholesale 
transactions and transmission of electricity in interstate commerce. 


Section 1.2: Key Participants 

The following is a list and brief description of electric industry participants relevant to the 
blackout of August 14: 


2 Alt but one of the smaller investor-owned utilities (and the two largest) are subsidiaries of much larger holding 
companies. 
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• American Electric Power Company (AEP) - a large public utility holding company 
that operates in eleven states. 3 Its subsidiary, Indiana Michigan Power Company, 
operates in the southwest comer of Michigan. 

• Consumers Energy Corporation - a large combined gas and electric utility, serving 

1 .7 million electric customers throughout most of the Lower Peninsula outside of the 
Detroit metropolitan area. It is a subsidiary of CMS Energy Company. 

• The Detroit Edison Company - the largest electric utility in Michigan, serving 2. i 
million customers in the Detroit metropolitan area. It is a subsidiary of DTE Energy 
Company and an affiliate of Michigan Consolidated Gas Company, which is a gas 
utility serving a similar territory. 

• East Central Area Reliability Council (ECAR) - a voluntary organization designed to 
augment electric reliability through coordinated planning and operation of its 
members’ generation and transmission facilities. The ECAR region includes Indiana, 
Kentucky, Michigan (Lower Peninsula only), Ohio, West Virgina, and small portions 
of four other states. 

• FirstEnergy Corporation - a public utility holding company with seven electric 
utilities and a transmission subsidiary operating in New Jersey, Ohio, and 
Pennsylvania. 

• Hydro One Networks, Inc. - one of four subsidiaries of Hydro One, Inc. It owns and 
operates 97% of the electric transmission lines in Ontario. 

• International Transmission Company (ITC) - an independent transmission company 
that owns and operates the transmission system formerly owned by Detroit Edison. 
ITC is a subsidiary of the investment firm Kohlberg Kravis Roberts & Co. 

• Michigan Electric Transmission Company (METC) - an independent transmission 
company that owns and operates the transmission system formerly owned by 
Consumers Energy. METC is a subsidiary of Trans-Elect, Inc. 

• Midwest Independent System Operator (MISO) - a regional transmission 
organization covering all or parts of Indiana, Illinois, Iowa, Kansas, Kentucky, 
Manitoba, Michigan, Minnesota, Montana, North Dakota, Ohio, Pennsylvania, South 
Dakota, and Wisconsin. Both ITC and METC are members of MISO. 

• North American Electric Reliability Council (NERC) - an umbrella organization 
formed in 1968 (after the New York blackout) to oversee the functions of ten regional 
reliability councils, including ECAR. 


3 In addition to Michigan, these are Arkansas, Indiana, Kentucky, Louisiana, Michigan, Ohio, Oklahoma, 
Tennessee, Texas, Virginia, and West Virginia. 


7 



384 


Ontario Independent Market Operator - a not-for-profit organization responsible for 
operating and regulating the wholesale electricity market in Ontario. 

PJM Interconnection - a regional transmission organization covering all or parts of 
Delaware, Maryland, New Jersey, Ohio, Pennsylvania, Virginia, West Virginia, and 
the District of Columbia. 


Section 1.3: Key Events from Michigan’s Perspective 

This section contains a summary of the key events leading to and following the blackout from 
Michigan’s perspective. It is not intended to include all events that could arguably have affected 
the blackout. Several organizations (FirstEnergy, ITC, METC, AEP, PJM, MISO, NERC, and 
the U.S./Canada Power Outage Task Force) have issued sequences of events. All of these 
sequences are generally consistent. This report draws from all of these in order to set forth what 
we consider to be the significant events, as viewed from Michigan’s perspective. 


Section 1.3.1: Thursday, August 14, 2003 

• 12:05:44 p.m. - AEP Conesville Unit 5, with a rating of 375 megawatts (MW), trips 4 
off-line. Most sequences begin with this event and it is included here only for 
completeness. In our opinion, this was not an event connected to the blackout. Since 
this unit had been down for two months, it was not expected to be producing 
electricity that day. The unit, which is located in central Ohio, was out of service for 
a short period and was back on-line at 3:20 p.m., nearly an hour before the blackout. 

• 1:14:04 p.m. - Detroit Edison Greenwood Unit 1, with a rating of 785 MW, goes off- 
line. Like the previous item, this event is included only for completeness, as it does 
not appear to be an event connected to the blackout. The unit experienced a boiler 
fuel trip at 12:45 p.m. The output of the unit was gradually reduced until it went off- 
line at 1 : 14 p.m. The unit, which is located north of the Detroit area, was brought 
back on-line and was producing about half of its rated capacity at the time of the 
blackout. However, other generators on the Detroit Edison system had been brought 
on-line to compensate for the capacity loss, so there was no overall reduction in 
output. The Michigan system was in balance prior to the last significant events 
before the blackout. 

• 1 :3 1 :34 p.m. - FirstEnergy Eastlake Unit 5, with a rating of 597 MW, trips off-line. 
This is arguably the first of many events that cumulatively led to the blackout. The 
unit, which is located in northern Ohio, along the shore of Lake Erie, was still off-line 
at the time of the blackout and the output of the unit had not been replaced. 

According to the testimony of Peter Burg (chairman and CEO of FirstEnergy) before 
the U.S. House of Representatives Committee on Energy & Commerce, the Eastlake 


4 The term “trip” refers to the automatic removal from service of a unit. 



Unit 5 tripped during the process of restoring the voltage regulator from manual to 
automatic control. According to Mr. Burg, this procedure was undertaken in order to 
stabilize reactive power output, so that the unit could later resume the requested 
voltage schedule. An ARS (automatic reserve sharing - a request for assistance to 
meet load when there is a loss on a utility’s system) was initiated for 595 MW from 
within the ECAR region. Mr. Burg indicated that they planned to supply some of this 
by cutting a 300 MW sale into PJM. 

3:05:41 p.m. - FirstEnergy Harding-Chamberlain 345 kilo-Volt (kV) line into 
northern Ohio trips, 

3:32:03 p.m. - FirstEnergy Hanna-Juniper 345 kV line in northern Ohio trips. It has 
been reported that this line disconnected because it sagged into a tree. 

3:41 :33 p.m. - AEP/FirstEnergy Star-South Canton 345 kV line in northern Ohio 
trips. AEP reports that this line opened due to high load and a phase to ground trip. 
AEP owns 0.69 miles of this line and FirstEnergy owns 33.42 miles. The 
U.S./Canada Power Outage Task Force reports that, with this and the prior two 
transmission lines disconnected, the effectiveness of the transmission path from 
eastern Ohio into northern Ohio was reduced. 

3:42 p.m. - Multiple smaller lines begin to trip open in northern Ohio. 

3:45:33 p.m. - AEP Canton Central-Tidd 345 kV line in northern Ohio trips open and 
reconnects 58 seconds later. According to AEP, the Canton Central-Colverdale 138 
kV circuit experienced six breaker operations in less than 3 minutes. This resulted in 
the 345/138 kV transformers being disconnected, isolating the 138 kV system from 
the 345 kV system. AEP owns 0.38 miles of the Canton Central-Cloverdale line and 
FirstEnergy owns 12.2 miles. 

4:06:03 p.m. - FirstEnergy Sammis-Star 345 kV line in northern Ohio opens, 
completely blocking the 345 kV path from eastern Ohio into northern Ohio. 

4:08:58 p.m. - AEP Gaiion-Muskingum River - Ohio Central 345 kV line in central 
Ohio opens due to high loading. 

4:09:06 p.m. - AEP East Lime-Festoria Central 345 kV line in central Ohio opens 
due to high loading, blocking transmission paths from southern and western Ohio in 
northern Ohio. 

4:09 p.m. - Six 138 kV lines from FirstEnergy Burger Unit substation open. 

4:09 p.m. - FirstEnergy Burger Units 4 and 5, with ratings of 150 MW and 135 MW, 
trip. 

4:09 p.m. - FirstEnergy Burger Unit 3, with a rating of 70 MW, is tripped manually. 
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• 4:09 p.m. - Power surges occur from southern Ohio, west to Indiana, north to western 
Michigan, east to the Detroit area, and south to northern Ohio. 

• 4:09:27 p.m. - Majestic-Tomkins interconnection between METC and ITC in central 
Michigan exceeds its emergency rating. 

• 4:09:27 p.m. - Bayshore-Monroe interconnection between ITC and FirstEnergy 
exceeds its emergency rating. 

• 4:09:23 to 4: 1 0:27 p.m. - Kinder Morgan plant in central Michigan, with a rating of 
500 MW, but loaded to 200 MW, trips. 

• 4:10 p.m. - FirstEnergy Harding-Fox 345 kV line in northern Ohio opens. 

• 4:10 p.m. - Twenty generating units along Lake Erie in northern Ohio, loaded to a 
total of 2, 1 74 MW, trip off-line. 

• 4:10:36 p.m. - METC Battle Creek-Oneida and Argenta-Tompkins lines trip. 

• 4: 1 0:38 p.m. - Pontiac-Hampton and Jewell-Thetford interconnections between 
METC and ITC trip, isolating the METC system from ITC. 

• 4:10:38 Midland Cogeneration Venture, loaded to 1,265 MW, trips. 

• 4: 10:38 p.m. FirstEnergy Perry-Ashtabula-Erie West 345 kV line trips, isolating 
northern Ohio from Pennsylavania. 

• 4: 10:39 p.m. - Power begins flowing in a giant loop from southern Ohio to 
Pennsylvania to New York to Ontario to Michigan to northern Ohio. 

• 4: 10:40 p.m. - Lemoyne-Majestic interconnection between ITC and First Energy 
trips. 

• 4: 1 0:40 p.m. - Allen Junction-Majestic-Monroe interconnection trips at Majestic and 
Monroe, which stops power flow to FirstEnergy’s Alien Junction interconnection 
with ITC. 

• 4: 10:41 p.m. - AEP Fostoria Central-Galion 345 kV line opens. 

• 4:10:41 p.m. - Lansing BWL interconnection with METC at Enterprise opens. 

• 4: 1 0:42 p.m. - Consumers Energy Campbell Unit 3, with a rating of 820 MW, trips. 

• 4: 10:42 to 4: 10:43 p.m. - Detroit Edison loses 1,863 MW of generation. 
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4:10:43 p.m. Monroe-Bayshore interconnection trips, isolating ITC from First 
Energy. 

4:10:43 p.m. - Consumers Energy Whiting Units 1 and 2 trip. 

4: 10:43 p.m. - Keith-Waterman 230 kV line, which connects Ontario with Michigan, 
opens. 

4: 10:40 to 4: 10:44 p.m, - Four transmission lines disconnect between Pennsylvania 
and New York. 

4: 10:45 p.m. - Transmission lines disconnect in Ontario and New Jersey. 

4: 1 0:52 p.m. - Oneida interconnection between Lansing BWL and ITC opens. 

4: 1 0:46 to 4: 1 0:55 p.m. - Transmission lines between New Y ork and New England 
(except for Connecticut) disconnect. 

4:10:50 to 4: 11:22 p.m. - Ontario interconnections with New York open. 

4:10:59 p.m. - Consumers Energy Whiting Unit 3 trips. 

4: 1 1 :05 p.m. - Lansing BWL’s Erickson plant trips. 

4: 1 1 :39 p.m. - All major Detroit Edison generation is off-line and most units have no 
station power. 

4: 1 1 :57 p.m. - Remaining transmission lines between Ontario and Michigan open. 

4:12:49 to 4:14:22 p.m. - Lansing BWL Eckert Station Units trip, leaving it with no 
generation on-line and no interconnections. 

4: 1 5 p.m. - The power outage is essentially complete. A total of 2.3 million 
customers of Consumers Energy, Lansing BWL, and Detroit Edison are without 
power. The area affected in Michigan is all of the Detroit Edison service territory, 
Consumers Energy customers located near the Detroit Edison service territory, and 
the cities of Lansing and East Lansing and other areas served by the Lansing BWL. 

4:30 p.m. - Michigan Public Service Commission (PSC) Staff arrive at the State 
Emergency Operations Center (SEOC) and initiate contacts with officials of Detroit 
Edison, Consumers Energy, Lansing BWL, and the U.S. Department of Energy 
(DOE) Emergency Operations Center in Washington, D.C. 
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• 5:00 p,m. - Michigan State Police (MSP) Emergency Management Division (EMD) 
activate SEOC. 

• 6:00 p.m. - Governor Granholm and her staff arrive at SEOC. 

• Evening - PSC Staff develop assessments, estimate recovery timeframes, and brief 
the SEOC staff, providing information on how to handle a power outage and on the 
safe use of generators for posting to the State government website. 

• 10:00 p.m, - The Governor makes a televised address on WKAR-TV to the State on 
the power outage (see Appendix A-l). 

• 10:00 p.m. - Consumers Energy reports 1 1 8,400 customers without power. 


Section 1.3.2: Friday, August 15, 2003 

• 12:44 a.m. - Marathon Refinery in Detroit is reported out-of-service. 

• 4:21 a.m. - Lansing BWL reports that power has been restored to all 98,000 affected 
customers. 

• 6:00 a.m. - Consumers Energy reports that 70, 1 00 customers remain without power. 

• 8:30 a.m. - Detroit Edison reports that 2,000,000 customers remain without power. 

• 9:15 a.m. - The Governor issues a Declaration of State of Emergency (see Appendix 
A-2) for the counties of Macomb, Monroe, Oakland, Washtenaw, and Wayne. 

• 9:16 a.m. - Executive Order No. 2003-10 (see Appendix A-3) is issued by the 
Governor suspending environmental specifications for gasoline required for use in 
southeastern Michigan. 

• 10:00 a.m. - At the request of the PSC Staff, Detroit Edison representatives arrive at 
the SEOC. 

• 10:00 a.m. - Detroit Edison reports 1,750,000 customers remain without power. 

• Noon - The Governor holds a press conference on the power outage and recovery 
efforts with J. Peter Lark, Chair of the PSC. 

• Early Afternoon - Consumers Energy reports 17,000 customers remain without 
power. 
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• Afternoon - PSC issues a press release urging Michigan citizens to take all 
reasonable steps to conserve energy in light of the devastating power outage 
throughout the northeast (see Appendix A-4). 

• Afternoon - At the request of the Michigan Department of Environmental Quality 
(DEQ), the U.S. Environmental Protection Agency (EPA) provides the State with 
enforcement discretion on environmental specifications for gasoline required for use 
in southeastern Michigan until August 22, 2003. 

• All day - PSC Staff continues monitoring restoration efforts and return to service of 
power plants not yet available to the grid. 

• Afternoon - Consumers Energy reports power restored to all 100,000 affected 
customers. 

• 10:00 p.m. - Detroit Edison reports that 500,000 customers remain without power. 


Section 1.3.3: Saturday, August 16, 2003 

• 6:30 a.m. - Detroit Edison reports power restored to all 2. 1 million affected 
customers. 

• PSC and Detroit Edison continue to monitor power loads and return to service of 
power plants not yet available to the grid. Customers are asked to reduce power 
usage, which prevented the need for a rotating blackout. 


Section 1.3.4: Sunday, August 17, 2003 
• The SEOC is deactivated. 


Section 1.3.5: Monday, August 18, 2003 

• The Commission issues an order in Case No. U-13859, directing an investigation into 
the extent, duration, and cause of the outage relating to Michigan customers. 


Section 1.3.6: Wednesday, August 20, 2003 

• J. Peter Lark, Chair of the Energy Advisory Committee, advises the Governor and the 
Energy Advisory Committee of an impending energy emergency due to dwindling 
gasoline supplies as a result of the power outage and damage to the Marathon refinery 
in Detroit. 
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• At the request of the DEQ, the EPA provides the State with enforcement discretion on 
environmental specifications for gasoline required for use in southeastern Michigan 
until September 3, 2003. 

• U.S. Energy Secretary Spencer Abraham and Canadian Minister of Natural Resources 
Herb Dhaliwal meet in Detroit and agree on an outline to be used by the U.S./Canada 
Power System Outage Task Force in its investigation. The Task Force will determine 
cause and effect of the outage. Three working groups addressing the electric system, 
security, and nuclear issues will be established to support the Task Force. 


Section 1.3.7: Thursday, August 21, 2003 

• 4:00 p.m. - Executive Order No. 2003-1 1 (see Appendix A-5) is issued by the 

Governor, which rescinds the Declaration of the State of Emergency and declares an 
Energy Emergency for the State of Michigan, due to loss of gasoline supplies as a 
result of the damage to the Marathon refinery and the temporary shutdown of other 
Midwest refineries supplying the Michigan market. 


• 4:02 p.m. - Executive Order No. 2003-12 (see Appendix A-6) is issued by the 

Governor, which continues in place the suspension of environmental specifications 
for gasoline required for use in southeastern Michigan. 


Section 1.3.8: Saturday, August 23, 2003 

• The Marathon refinery in Detroit resumes production of petroleum products, 

including gasoline, that meet the specifications required under air quality rules for 
southeast Michigan. The refinery, which was shutdown for eight days following the 
outage, lost nearly 500,000 barrels of petroleum product production, roughly half of 
which was gasoline (about 1 1 million gallons). This is equal to about 3 percent of the 
projected statewide gasoline demand in August. However, the concentration of the 
lost supply in southeast Michigan made the area’s shortfall larger than this figure 
suggests for that region. 


Section 1.3.9: Wednesday, August 27, 2003 

• Secretary Abraham and Minister Dhaliwal, as Co-Chairs of the U.S./Canada Power 
System Outage Task Force, announce the membership of the three working groups 
that will support the Task Force. Participating from Michigan on the Electric 
Systems and Nuclear Power Working Groups will be J. Peter Lark, PSC Chair. 
Participating on the Security Working Group will be Colonel Michael C. McDaniel, 
Assistant Adjutant General for Homeland Security for Michigan. 
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Section 1.3.10: Thursday, August 28, 2003 

• At the request of DEQ, the EPA provides the State with enforcement discretion on 
environmental specifications for gasoline required for use in southeastern Michigan 
until September 15, 2003, at which point the summer environmental specifications 
are no longer in effect. 


Section 1.3.11: Wednesday, September 3, 2003 

• Governor Granholm and PSC Chair Lark testify before the U.S. House of 
Representatives Committee on Energy and Commerce on the power outage. 


Section 1.3.12: Tuesday, September 30, 2003 

• Executive Order No. 2003-16 (see Appendix A-7) is issued by the Governor 
rescinding the Declaration of Energy Emergency for the State of Michigan, 
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PART II 

ELECTRIC TRANSMISSION 


Section 2.1: Scope of the Investigation 

Although the electric transmission events that led to the blackout were regional, and perhaps 
national in scope, the focus of this investigation has been on the impact to Michigan. In the 
order initiating this inquiry, we concluded “that there should be a Michigan-specific 
investigation into: (a) the extent, duration, and causes of the outage relating to Michigan 
customers; (b) the reaction of Michigan electric utilities and transmission grid operators ... to the 
outage and the events preceding it: (c) a comprehensive assessment of the power restoration 
efforts by Michigan companies; and (d) recommendations designed to prevent future 
disruptions.” 

The central geographic focus of the blackout was on the “Lake Erie Loop.” The Lake Erie Loop 
is that portion of the eastern interconnection of the electric grid that runs around Lake Erie, most 
directly impacting the border states of Michigan, Ohio, Pennsylvania, New York, and the 
Canadian Province of Ontario. However, as evidenced by the blackout, other states were also 
affected as a result of the extensive interconnectedness of the electric grid running throughout the 
entire eastern interconnection, which covers much of the eastern portion of the country east of 
the Rocky Mountains. 

Our overall investigative focus has been prospective. Although we examined factors 
contributing to the cause of the blackout, the principal objective was to identify reliability issues 
and concerns that were uncovered from the investigation that can provide guidance on ways to 
strengthen overall grid reliability. Although a comprehensive array of options has been included, 
our focal point of emphasis was regional grid coordination. 

It is important to emphasize that this report does not attempt to establish the “root cause” of the 
blackout or directly assign blame. That responsibility is best relegated to the U.S./Canada Power 
System Outage Task Force, which set determination of root cause among its principal objectives. 
That Task Force is much better positioned to address the root cause question. Clearly, compared 
to the Commission, it has significantly greater access to information and the substantial resources 
needed to dig deeply into causal issues. Likewise, the scope of the international investigation is 
much broader geographically than the Michigan study, which is concentrated primarily in and 
around Michigan. Finally, as discussed more fully later in this report, we conclude that the 
events leading up to the blackout occurred in Ohio, which is beyond the purview of this 
Commission. 

In conducting this investigation, the following questions were addressed: 

• What happened? 

• How was Michigan impacted? 

• What are the lessons learned? 

• What can be done to prevent a blackout like this from recurring in the future? 
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Section 2 . 2 : Study Approach 

In conducting the investigation, we utilized a wide variety of information sources. The following 
are the principal data sources relied upon: 

(1) Published information from a wide variety of sources was utilized. In the wake of the 
blackout, numerous reports, news releases, and articles were published by a variety of sources, 
both public and private. Among the information sources reviewed were professional reports, 
analyses from industry and academic experts, testimony from State and Congressional hearings, 
trade press articles, governmental statements and press releases, and many newspaper and 
magazine articles. Although often redundant, the flood of information following the blackout 
proved quite helpful in piecing the events together into a comprehensive picture. 

(2) Interviews with key participants charged with grid reliability were a crucial source of 
information to the investigation. Key to the study investigating regional coordination aspects of 
the blackout was information obtained from extensive interviews with various individuals with 
reliability and operational responsibility throughout the region. Included in these interviews 
were representatives from: 

• Michigan’s two major transmission companies operating the transmission grid in the 
lower peninsula - the International Transmission Company and the Michigan Electric 
Transmission Company. 

• The American Electric Power Company transmission reliability coordination and 
transmission operations personnel. 

• The Michigan Electric Coordination System - which oversees the local reliability 
control areas for ITC and METC covering most of the Lower Peninsula including the 
service territories of Detroit Edison and Consumers Energy and is the control center 
where certain reliability functions are jointly managed under the decentralized 
reliability system within the Midwest Independent System Operator. 

• The Midwest Independent System Operator. 

• The PJM Interconnection - the regional transmission organization for the Mid 
Atlantic states and some key Midwestern utilities including AEP. 

• The New York Independent System Operator (New York ISO). 

• The Ontario Independent Market Operator (Ontario IMO). 

• The Federal Energy Regulatory Commission. 

(3) Primary data was also very important to the investigation. Review of reliability and 
operations control room telephone conversations from all the above listed organizations, except 
the Ontario IMO and the New York ISO, was instrumental to examination of grid management 
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issues and problems, especially communications coordination among personnel from the various 
reliability controllers within the region. Although much of the relevant data from these 
communication links had already been reported through the media, the opportunity to directly 
examine the actual conversations that took place helped to empirically verily the accuracy of 
those reports. The additional detail provided by the full text of those conversations also 
significantly enriched our understanding of what took place in the reliability centers within the 
region affected by the blackout. 

Although many and varied sources of information were carefully evaluated and cross-checked to 
ensure reliability in the conduct of this investigation, in the final analysis, this report’s findings, 
conclusions and recommendations largely reflect the professional expertise and judgment of the 
Commissioners and our Staff. Staff assignments to perform the study were made with that in 
mind, coupled with an appreciation of the importance of the blackout investigation. The 
expertise and analytical capability of the investigation team encompasses considerable academic 
training, along with extensive electric industry knowledge and professional experience. 


Section 2.3: Events of August 14, 2003 

As previously mentioned, this investigation does not intend to identify the root cause of the 
blackout. Rather, it identifies the significant events leading up to the blackout as an aid to 
analyzing what steps are needed to help prevent a reoccurrence. 

A list of specific events leading up to the blackout is presented in Section 1.3. A review of this 
timeline indicates that all of the events in the two and one-half hours preceding the power surges 
that occurred at 4:09 p.m. involved FirstEnergy or AEP facilities in Ohio. In addition, 
FirstEnergy’s Davis Besse nuclear plant had been out-of-service for some time. It appears that, 
with the Davis Besse nuclear plant off-line, the tripping of Eastlake Unit #5 was a major event in 
the northern Ohio region. FirstEnergy was left in a precarious position as far as meeting its load 
on that day. Power had to come from other sources in order to meet the requirements of the 
FirstEnergy system. FirstEnergy did initiate a request for automatic reserve sharing and MECS 
did respond to this request. Nonetheless, lines in Ohio soon began to open. Although 
information provided by M1SO and FirstEnergy indicate that the first lines that opened were not 
loaded to capacity, it appears that the opening of those lines quickly contributed to the evolving 
problem. As the number of lines opening in Ohio increased, there were fewer and fewer paths 
available to serve the demand coming from northern Ohio. According to data from Michigan 
transmission operators and the MISO, at about 4:09 p.m., over 2,000 MW of power was 
suddenly pulled from west to east through Michigan and into northern Ohio. Voltage on the 
Michigan grid became unbalanced. Seconds later, the flow suddenly reversed and over 2,000 
MW were pulled into Michigan from the east, again attempting to reach load in northern Ohio. 
There can be little doubt, based on the sequence of events and magnitude of the power flow 
reversals, that disturbances in Ohio led to the blackout. Evidence strongly suggests these 
disturbances led to an internal load balancing collapse of the FirstEnergy system, which 
ultimately precipitated the blackout. 
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All of the people interviewed for this investigation agreed that August 14, 2003 was a normal 
summer day. However, there were some indications that the day was turning out to be hotter and 
more humid than had been predicted. Early in the day PJM issued a “high load voltage 
warning.” In our interviews with Michigan and other state transmission operators, no specific 
supply problems were noted and the overall supply situation for the region was very adequate. 
Transmission operators and reliability coordinators (RCs) had been communicating normally 
during the day. MISO had been working with Cinergy 5 to relieve congestion in its territory. 

In the very last minutes preceding the blackout, Michigan operators saw large flow swings of 
over 2,000 MW. As a result of the declining voltage, the protective devices on three Michigan 
power plants tripped. According to ITC records, within seconds it was in full voltage collapse 
with over 30 transmission lines opening. METC and ITC connections separated as the relays 
responded to programmed settings. Several more plants in DTE’s service area tripped. Most of 
ITC’s system was blacked out, only a few scattered areas still had electricity. METC’s system 
remained largely intact, although there were some scattered outages. 

The Greenwood power plant outage on the Detroit Edison system was not a factor contributing 
to the blackout. The Greenwood plant experienced a fuel trip at approximately 1:14 p.m. and 
was returned to service in less than 45 minutes at about 1:57 p.m. During the Greenwood outage 
power from other generation facilities on the Detroit Edison system was brought on-line for 
replacement of the lost power from the Greenwood generation unit. The facility was fully 
resynchronized to the system more than two hours before the blackout. 

The first event that MISO observed was the opening of the FirstEnergy Hanna-Juniper line at 
3:32 p.m. 6 MISO was unaware that the FirstEnergy Harding-Chamberlin line had opened at 3:06 
p.m. because MISO only follows “key facilities.” At the time of the blackout, the Harding- 
Chamberlin line had not been identified by FirstEnergy as a key facility. 

According to the MISO telephone transcripts, MISO called First Energy at 3:43 p.m. and 
questioned FirstEnergy about the Hanna-Juniper line. The FirstEnergy operator was not able to 
respond to MISO's questions and said that he didn’t know, that he would have to take a look. 
MISO requested that FirstEnergy call it back. At 4:04 p.m., FirstEnergy called MISO and stated 
that they had some problems. The FirstEnergy operator still seemed unsure about exactly what 
was happening. The operator lists a number of lines that are “off’, the Eastlake Plant unit that 
had gone off-line earlier in the day and the Perry plant that was “having a hard time maintaining 
voltage”. The FirstEnergy operator then asks MISO what it has going on. When MISO responds 
that FirstEnergy Hanna-Juniper line is open, the FirstEnergy operator questions that. MISO 
responds that it had discussed this with FirstEnergy earlier. The FirstEnergy operator states that 
they have “no clue” and the computer is "giving us fits.” A FirstEnergy control room operator 
told a MISO technician minutes before the blackout, “We don’t even know the status of some of 
the stuff around us.” The MISO operator states that MISO thought FirstEnergy was trying to 


5 Cinergy Corporation provides natural gas and electric utility service in Indiana, Kentucky, and Ohio. 

6 FirstEnergy was not a member of MISO on August 14, although MISO was its reliability coordinator. On October 
1, 2003, American Transmission Systems, Inc., the transmission subsidiary of FirstEnergy became part of MISO 
through GridAmerica LLC. GridAmerica, a subsidiary of National Grid USA, manages the transmission system of 
FirstEnergy, Northern Indiana Public Service Company, and Ameren Corporation. 
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figure this out. The FirstEnergy operator says they are trying to and then asks if the MISO sees 
anything else going on around them. The MISO mentions that Cinergy had some lines opening 
earlier in the day, but the FirstEnergy operator responds that that shouldn’t have affected them. 
The transcript ends in mid-sentence. Based on the MISO telephone transcripts, MISO did not 
talk to any other control areas about the situation at FirstEnergy. Thus, MISO was unable to 
communicate effectively with its members, and FirstEnergy was unable to communicate 
effectively with its reliability coordinator, MISO. 

It is important to note that Peter Berg (the CEO of FirstEnergy), in his Congressional testimony, 
stated that although First Energy was experiencing problems with its energy management 
computer system, MISO had information about FirstEnergy’s system and could have been used 
as a backup. As is discussed above, MISO did not have access to all of FirstEnergy’s 
information, in that MISO only had information on the key facilities that had been identified by 
FirstEnergy. However, if FirstEnergy was relying on MISO to provide back up information, 
then it makes sense that FirstEnergy should have contacted MISO to request that back up. Based 
on our review of the MISO telephone transcripts, there is no indication that FirstEnergy called 
MISO to report their dilemma until 4:04 p.m. This was several minutes after MISO had 
contacted FirstEnergy to discuss the Hanna-Juniper line. During the call that was initiated by 
MISO, FirstEnergy did not describe any problems. It should be noted that FirstEnergy did not 
have any problem contacting MISO after the event. The MISO transcripts show a number of 
calls from FirstEnergy to MISO asking for assistance in the restoration efforts. 

AEP and PJM also talked to FirstEnergy and were talking to each other. Based on our review of 
these calls, FirstEnergy provided less than enlightening information. Both AEP and PJM appear 
to have gotten the impression, either directly or indirectly, that FirstEnergy was having computer 
problems. AEP also called PJM when AEP saw that the South Canton-Star line was becoming 
overloaded. On the telephone transcripts that we reviewed, AEP initially thought that the South 
Canton-Star line was a FirstEnergy line. After AEP determined that the line was jointly owned, 
AEP and PJM began discussing the issuance of a Transmission Loading Relief (TLR) 7 
procedure. Before the TLR could be issued, the South Canton-Star line opened. As a result of 
that line opening, PJM called FirstEnergy to discuss the contingency line (Sammis-Star), a line 
that belonged to First Energy. First Energy was unable to provide any meaningful assistance to 
PJM and told PJM to talk to AEP. 

The two reliability coordinators for the Midwest utilities, MISO and PJM, did call FirstEnergy 
shortly before the blackout to try and discuss potential problems they were observing on the 
FirstEnergy system. As described in the detail of the calls discussed above, the response from 
First Energy was confused, at best. Earlier in the day FirstEnergy had called MISO to report a 
discrepancy in the outage report on the time that the Eastlake plant went out of service. 
FirstEnergy indicated during that call, which occurred at 2:24 p.m. (over 50 minutes after 
Eastlake went out of service), that they did not know why the plant had tripped. Based on the 
review of telephone transcripts that we reviewed, FirstEnergy did not contact MISO concerning 
any other problems that FirstEnergy was having that afternoon until 4:04 p.m. - only 5 minutes 
before the cascading blackout events began. 


7 Transmission Loading Relief is a process that allows reliability coordinators to curtail transmission service. 
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Immediately after the blackout, grid restoration efforts were commenced. MISO kept a 
telephone line open so that all parties involved could communicate with each other and assist 
each other with restoration efforts. The code of conduct, which requires independent functioning 
of transmission operators and affiliate wholesale marketers except in an emergency, was 
appropriately suspended to permit communication channels to be fully opened. Overall the grid 
restoration went relatively smoothly throughout the region. 


Section 2.4: Analysis 

Section 2.4.1: Electric Industry is in Transition 

Understanding critical changes shaping the electric industry is important to understanding how 
the events of August 14 unfolded and what can be done to reduce the likelihood of a recurrence. 

The electric industry looks very different today than it did when it first emerged at the turn of the 
century. Most notably, the nation’s electricity delivery system has grown and evolved from a 
very local, insular, largely self-sufficient network to one that is highly interconnected and 
regionally interdependent. The August 14 111 blackout was a clear reminder of just how 
interdependent the electric grid has become. 

It is somewhat ironic that grid interconnections, the mechanism that physically enabled the 
blackout to spread through such a wide area, were originally established to improve reliability. 

In the electric industry’s infancy the transmission network served exclusively as the mechanism 
used to move electric power from generating plants to customer load within a local utility 
company’s service territory. However, as the industry matured, companies soon discovered that 
by linking with neighboring local utility companies they could assist each other in times of 
emergency and improve overall reliability of service to their customers. Companies were 
mutually benefited through sharing of supply reserves. 

Interconnecting utility transmission facilities over larger geographical areas, while increasing 
overall reliability, also created dependence among independently operated utility systems. This 
relationship, as illustrated by the blackout, means connected utility systems are now vulnerable 
to events occurring on other utility systems linked to the chain. While no credible argument can 
be made advocating that grid interconnection is not beneficial to overall industry reliability 
improvements, the tradeoff is that disturbances on one system, if not contained, can cascade 
throughout the interconnected grid as they did on August 14. Thus, while grid interconnection 
can be expected to reduce the probability and duration of an outage, it also introduces the 
possibility of a significant event or a series of smaller events on one system triggering a large 
outage. The key challenge going forward is to take steps to localize disturbances to prevent 
them from cascading. 
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Section 2.4.2: Electric Industry is Responding to Competitive Pressures 

Over the past 20 years real (inflation adjusted) electricity prices paid by consumers have 
significantly decreased. Since 1990 real electric prices have dropped nearly 12% nationally, 
almost 18% in the ECAR States, and 21.5% in Michigan (see Figure 2.1). The largest declines 
are in the Midwest where manufacturing demands for efficiency are most intense. Among the 
more significant factors driving these recent efficiency gains is the increasingly competitive 
wholesale electricity market that has emerged during this period. 
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FIGURE 2.1 


Average Retail Prices of Electricity, 1990-2001 

(Inflation-Adjusted Cents per Kilowatthour) 


Michigan Total 

Regional Total 

National Total 

Year 

Real' 

! % Increase 

Real ■ 

| % Increase 

Real ’ j % Increase 

1990 

8.8852 

0.0% 

7.1982 

0.0% 

8.2197 

0.0% 

1991 

8.7006 

-2.1% 

7.0457 

-2.1% 

8.1440 

-0.9% 

1992 

8.5273 

-4.0% 

6.8218 

-5.2% 

8.0425 

-2.2% 

1993 

8.2153 

-7.5% 

6.7074 

-6.8% 

7.9704 

-3.0% 

1994 

7.9751 

-10.2% 

6.5380 

-9.2% 

7.7687 

-5.5% 

1995 

7.7656 

-12.6% 

6.3838 

-11.3% 

7.5860 

-7.7% 

1996 

7.6678 

-13.7% 

6.2685 

-12.9% 

7.4102 

-9.8% 

1997 

7.4727 

-15.9% 

6.1365 

-14.7% 

7.2757 

-11.5% 

1998 

7.4391 

-16.3% 

6.1672 

-14.3% 

7.0685 

-14.0% 

1999 

7.3782 

-17.0% 

6.0912 

-15.4% 

6.8949 

-16.1% 

2000 

7,2517 

-18.4% 

5.9813 

-16.9% 

6.9463 

-15.5% 

2001 

6.9788 

-21.5% 

5.9254 

-17.7% 

7.2600 

-11.7% 

1 In inflation-adjusted (2001) dollars, calculated by using Gross Domestic Product deflators. Source is: 

Budget of the United States Government, 

Table 10.1 

—Gross Domestic Product and Deflators 

Used in the Historical Tables: 1940-2005, 

(wk4 or xls). 




Prepared by: Competitive Energy Division, Michigan Public Service Commission, September 2003. 


Data Source: Energy Information Administration, U.S. DOE, Electric Power Annual 2001 Table 7.4, Annual 

Energy Review 2001 







Table 8.6, Form EIA-861 Database 





http://www.eia.doe.gov/cneaf/electricity/epa/epat7p4.html 




EIA sales_states.xls, evenue_states.xls 





http://www.eia.doe.gov/cneaf/electricity/epa/epa_sprdshts.html 




Average Annual Electricity Retail Prices 
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In recent years demands on the electric industry to power our nation’s growing economy and 
satisfy consumer needs has provided the world standard for safety, convenience, and comfort at 
affordable prices are unprecedented. By and large, the industry has stepped up to the challenge. 
Since 1990 demand (nationally, regionally and in Michigan) has increased by about 30% (see 
Figure 2.2). Not only has this significant increase in demand been met, it has been accomplished 
at lower cost to consumers. Nevertheless, increased demand is a factor stressing the grid’s 
reliability capability. Industry restructuring has also placed stress on grid reliability. Opening 
the electric industry to competition has introduced new participants (such as independent 
transmission companies, independent generators, power marketers and energy traders) and 
expanded the number of power exchange transactions as well as the distances power is 
transported. Growing pains are also evident as the transmission network struggles to 
accommodate the sudden and significant increase in power transactions over extended 
geographic areas. Cost-effective investment is needed to alleviate critical bottlenecks and 
expand transport capability in some areas. Likewise, significant attention to measures to 
improve regional grid coordination and management is essential to both electric grid reliability 
and efficiency improvements. 

Competitive electric industry restructuring, while a factor stressing the grid within the Lake Erie 
loop, is not the cause of the August 14 th blackout. Electrons moving through the grid follow the 
laws of physics, not economics. The grid is indifferent to whether electricity moving through the 
wires is produced and sold under a market structure that is competitive, regulated or a hybrid of 
the two. Our investigation of the August 14 th blackout found no basis to dispute this conclusion. 
However, what we did discover is that in the current transition, the “rules of the road” defining 
the relationship between reliability and efficiency need considerable revision if both reliability 
and efficiency are to effectively coexist in today’s complex restructuring industry. The challenge 
is to make adjustments to the grid and its management that will continue to make the two critical 
objectives complementary. 
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FIGURE 2.2 


Michigan, ECAR, National Electric Peak Demand 

(Non-coincident Peak in Megawatts) 


Year 

Summer Michigan 

Summer ECAR 

Summer U.S. 

CE 

DTE 

_ , , % Increase 

Total 

over 1990 

_ , , % increase 

Total 

over 1990 

_ , , % Increase 

Totai 

over 1990 

1990 

5,891 

9,032 

17,905 

0.0% 

79,258 

0.0% 

545,537 

0.0% 

1991 

6,084 

8,980 

18,247 

1.9% 

81,539 

2.9% 

551,705 

1.1% 

1992 

5,939 

8,704 

17,759 

-0.8% 

78,550 

-0.9% 

548,707 

0.6% 

1993 

6,226 

9,362 

18,758 

4.8% 

85,930 

8.4% 

580,753 

6.5% 

1994 

6,502 

9,684 

19,386 

8.3% 

87,165 

10.0% 

585,320 

7.3% 

1995 

7,158 

10,049 

20,621 

15.2% 

92,819 

17.1% 

620,249 

13.7% 

1996 

7,167 

10,377 

21,032 

17.5% 

90.798 

14.6% 

616,790 

13.1% 

1997 

7,315 

10,305 

21,224 

18.5% 

93.492 

18.0% 

637,677 

16.9% 

1998 

7,246 

10,704 

21,670 

21.0% 

93,784 

18.3% 

660,293 

21.0% 

1999 

7,460 

11,018 

22,223 

24.1% 

99,239 

25.2% 

681,449 

24.9% 

2000 

7,306 

10,730 

21,994 

22.8% 

97,557 

23.1% 

685,816 

25.7% 

2001 

8,289 

11,860 

24,417 

36.4% 

102,161 

28.9% 

709,166 

30.0% 


Total U.S, noncoincident peak demand is for conligous U.S. without Alaska or Hawaii. 

Prepared by: Michigan Public Service Commission 

Sources: May 2003 Michigan Electric Sales Forecast, 1990-01, Energy Information Administration, U.S. DOE, 

Form EIA-861 Database. ECAR, National data source: EIA Electric Power Annual Table 3. t , EIA Annual 
Energy Review 2001 Table 8.8, Form EIA-41 1 

see http://www.eia.doe.gov/cneaf/electricity/epa/epat3p1 .html 
http://www.eia.doe.gov/emeu/aer/elect.html 
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Section 2.4.3: Grid Reliability Responsibility 

The North American Electric Reliability Council (NERC) has operated since 1968 as a voluntary 
organization with the goal of ensuring that the electric transmission system is reliable, adequate 
and secure. NERC does this through two primary tools: planning standards and operating 
policies. Planning standards are used in the development of new transmission systems. The 
operating policies are made up of standards, requirements and guides with the intent that no 
system will impact the integrity of any of its interconnected systems. The primary responsibility 
for actually carrying out the operating policies is on the control area. A control area, as defined 
by NERC, is an electric system or systems, bounded by interconnection metering and telemetry, 
capable of controlling generation to maintain its interchange schedule with other control areas 
and contributing to frequency regulation of the Interconnection. 

Reliability coordinators (RC) act under the auspices of NERC and are responsible for ensuring 
the operational reliability of the interconnections (the facilities that connect two systems or 
connect a system to a non-utility generator). The reliability coordinator plans for next day 
operations, analyzes current day operating conditions and implements the transmission loading 
relief (TLR) procedure. The TLR procedure is a key component of the reliability coordinator’s 
job since it is intended to relieve overloaded conditions on a transmission system. In a critical 
situation, the reliability coordinator has the authority to direct control area operators in its 
reliability area to do whatever is necessary to relieve the situation. The reliability coordinator is 
also required to advise all control areas in its reliability area and all other reliability coordinators 
if an emergency exists in its area. This notification can be done through the RCIS (Reliability 
Coordinator Information System), a computer program that posts information to a central web 
site that can be accessed by all reliability coordinators and control area operators. 

Enforcement of the standards, in practice, has been accomplished through voluntary compliance. 
Currently, NERC portrays its compliance process as mandatory, but not enforceable. NERC, 
through its regions, conducts compliance reviews and issues enforcement advisories. The 
enforcement advisory can be in the form of an award for good performance or a simulated 
sanction or penalty. NERC uses a simulated sanction or penalty because it does not have the 
authority to issue an actual penalty. Under the NERC RC Procedures, all Control Areas must 
comply with a directive from the RC. The 2002 NERC Compliance Enforcement Program 
Report indicates that there were 444 violations involving operating measures in 2002. These 
violations resulted in $9 million of simulated sanctions, which could have been actual penalties 
had penalty provisions been in place and enforced. 

FERC is the regulatory agency that oversees the interstate transmission of electricity. The 
agency regulates the economic aspects of interstate transmission and wholesale sales of 
electricity. FERC states that it may not have regulatory authority over the construction of 
transmission or generation facilities, except for hydroelectric facilities. 

Regional grid coordination and management must be strengthened through strong and properly 
configured RTOs. Effective regional grid reliability coordination and management is key to 
improving reliability on an increasingly interdependent transmission grid. In the Midwest, this 
should be accomplished by mandatory RTO participation by transmission owners and operators. 
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Ways to quickly resolve parochial interests and move toward effective regional solutions to 
regional reliability issues must be forged. 

Mandatory RTO participation, where FERC deems it necessary, is essential. RTO development 
in the Midwest underscores the importance of having a single RTO in the region. Serious 
configuration and seams problems are stifling RTO development in the Midwest region and 
threaten the region’s ability to effectively manage reliability. Failure to correct these problems 
could deliver a fatal blow to effective RTO development in the Midwest. The best-designed 
reliability standards are of little value if they are not uniformly and widely enforced. The system 
is only as strong as its weakest link. Allowing transmission operators within an interconnected 
grid to opt out of the RTO or to dictate the terms of RTO participation, as often occurs under the 
present voluntary system where competing RTOs in the Midwest vie for prospective members, is 
a recipe for disaster. On August 14 th , no one chose to be a victim of the blackout. No one should 
likewise be excused from participation in those efforts necessary to prevent a future occurrence. 


Section 2.4.4: Effect of Competition 

Although a factor, electric industry restructuring did not cause the blackout. Primarily an 
outgrowth of change, the impact of competition on reliability is a factor that can be effectively 
managed. However, the relationship between reliability and efficiency must be clearly 
established such that reliability reigns supreme where conflicts surface between reliability and 
efficiency. This fundamental principle must be clearly established, understood and rigorously 
enforced. 

Reliability enforcement decisions should be placed in the hands of an independent party. Placing 
authority or any significant control over grid reliability decisions in the hands of companies with 
a commercial interest at stake must be prevented. Removing the temptation to avoid 
implementation measures that may be needed to localize or contain a disturbance is a must. 

There is no question that a reliability coordinator’s decision to take action to shed load during a 
disturbance must take precedence over the commercial impact of the decision. We did not find 
any evidence to conclude that such a conflict played a role on August 14. However, we note that 
both companies experiencing disturbances, FirstEnergy and AEP, still play a significant role in 
the execution of reliability coordination monitoring and decision-making within their respective 
RTOs. Both AEP and FirstEnergy are vertically integrated utility companies with a financial 
stake in the commercial operations for energy sales and delivery (transmission and distribution) 
to their utility customers. This potentially conflicting responsibility is problematic. 


Section 2,4.5: Grid Investment 

Strategic grid investment is needed to effectively accommodate industry changes necessary to 
continue to meet high industry standards for reliability and efficiency. According to the NERC’s 
Reliability Assessment 2002-2001 issued in October 2002, the number of TLRs requiring 
curtailment of firm transactions increased from zero in 1998, to I in 1999, 7 in 2000, to 16 in 
2001, to 18 in the first six months of 2002. This suggests a transmission grid that is under 
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increasing pressure to meet the demands placed upon it. That NERC report indicated that 
“portions of the transmission systems are reaching their limits as customer demand increases and 
the systems are subject to new loading patterns resulting from increased electricity transfers.” 8 

Customers expect and deserve reliable and competitively priced electricity. With that in mind, 
grid investment to expand and upgrade the transmission system should be cost-effective so that 
investment is strategically targeted to cost-effective congestion and growth needs. According to 
the NERC report, only 155 circuit miles of new transmission is planned for the ECAR region 
through 2006 out of an existing base of 16,207 miles. Moreover, no new transmission is planned 
in ECAR during the 2007-20 1 1 time period. This equates to an increase in transmission lines of 
less than 1% over the next 8 years. For comparison, in the United States as a whole, 
transmission lines are expected to grow by approximately 6% during the same period. This 
suggests that consideration should be given to additional transmission lines in this region. 


Section 2.4.6: Grid Reliability 

Grid reliability coordination and management must be accomplished on a regional basis. The 
blackout served as a wake-up call to accelerate regional coordination of all aspects of grid 
development and operation. Parochial utility and state interests must give way to more effective 
regional grid planning, operation and management to advance grid reliability objectives. In 
particular, actions must by expeditiously undertaken to establish strong and effective RTOs 
throughout the Midwest, Mid-Atlantic, and New England regions, which have become 
increasingly interdependent and generally supportive of RTO development. 


Section 2.4.6.1: Key Facilities and Other Information 

The RCs need to see the big picture, including all key facilities. The MISO is already addressing 
the use of key facilities. FirstEnergy, for example, has gone from 35 to 90 key facilities that are 
available to the MtSO for detailed on-going review. ITC and METC have indicated that they 
have made all of their data available to the MISO. Other control areas that are members must 
also make all of their data available to the MISO. The RCs need to see into the areas that are 
connected to them. The MISO and PJM had the ability to see some of this but could not see all 
of the pertinent areas. The RCs should work together to determine what information they need 
in order to bring the big picture into focus. The interconnections that are common all across the 
country are making the ability to see a more expansive, comprehensive and timely view of the 
transmission system increasingly essential to maintaining reliability. System monitoring 
capability must be state of the art to successfully accomplish this demanding task. 


8 Quotation is from page 20 of the report. Interestingly, that page begins with the statement that: “North American 
transmission systems are expected to perform reliably in the near future.” 
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Section 2. 4. 6.2: Reliability Control Operators 

There are indications that more or better-trained reliability control operators may be needed. 
There should be a sufficient number of well-trained operators in the control room to handle any 
number of emergencies that might arise. On the MISO transcripts, reliability operators spent 
considerable time dealing with an emergency on the Cinergy system. While that event did not 
directly impact the events leading up to the blackout, it did appear that the MISO operators were 
having some difficulty keeping up with the numerous functions that needed their attention that 
day. In fact, the MISO’s state estimator, which is one of its system monitoring tools, was not 
operating properly for several hours, and it appears that in the confusion, no one noticed. The 
control centers must be properly staffed and the operators properly trained, so that in an 
emergency situation adequate resources are available to respond to the emergency and to the 
ongoing system operation. 

Control room equipment and operations should be thoroughly examined to ensure they are state 
of the art. The grid is a most complex and interdependent network. Individuals responsible for 
monitoring the grid and deciding when action is appropriate to prevent outages or isolate systems 
to prevent outages from cascading face a daunting challenge. Reliability control room operators 
must be properly trained, equipped, and provided with appropriate protocols to carry out these 
most important responsibilities. 

Good communications are critical. The transmission grid is becoming more and more 
interconnected. This is not going to change; in fact, the trend is likely to increase. The 
interconnections provide for increased reliability and assistance during critical times. A 
communications chain needs to be established and adhered to by all of the entities. In the past, a 
control area may have only had to call its neighbor, and all of the operators probably knew each 
other by name. Now, the control area has to determine whom it should call. On the list is the 
RC (if the RC is different than the control area), the transmission operations centers (generally 
the utility or independent transmission company that actually operates the controls), and the 
neighbors (control areas that are connected). On the day of the blackout, there was confusion. 
AEP was calling FirstEnergy directly, AEP was calling PJM, PJM was calling FirstEnergy, PJM 
considered calling MISO but did not have time before events started to cascade. Another 
problem seemed to be that callers did not always know who they were talking to. There were 
sometimes two or three exchanges on “who is this”, “who are you with”, “what is your 
responsibility”. A communications chain would also ease this problem, because each link in the 
chain would know who they are supposed to call and who is calling them. 

Another significant part of communications is the communication system itself. The telephone is 
still being used to a large extent to communicate with others regarding reliability concerns. The 
whole system should be reviewed and the expanded use of computers should be explored. With 
e-mail and instant messaging now available, communications could be put in writing, which 
could provide clarification and consistency. Systems that are already in place, such as the RCIS, 
should be utilized more extensively. The phone system itself should also be examined to provide 
better and more reliable phone service. Based on our interviews with transmission operators, the 
phone service at the Michigan control center was sporadic after the blackout. 
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Communication protocols should be developed to more effectively address regional reliability 
coordination needs. Protocols directing appropriate procedures for information exchange among 
reliability coordinators are much needed. Much confusion was evident in reliability coordination 
control centers on August 14 th regarding communication protocol. 

Staffing qualifications, training and levels should be reviewed. Although our investigation did 
not uncover any specific staffing problems, our review of control room conversations and 
interviews with industry representatives led to the conclusion that operators may have been 
stretched to their limits on August 14 th . In light of significant industry changes and growing 
electric grid demands, control room staffing levels and training needs should be carefully 
examined. We recommend that a comprehensive investigation of reliability control room 
staffing issues be undertaken by NERC with FERC oversight. Any deficiencies identified from 
the investigation should be promptly corrected. 


Section 2.4.6.3: Reliability Standards 

Mandatory reliability standards are needed. Many of the industry expert testimonies, articles and 
reviews of the blackout, along with responses to our interview questions, strongly urged the 
implementation of mandatory reliability standards. This report concurs with that conclusion. 

The requirement that standards be applied, with a resultant penalty if they are not, is a critical 
factor in reliability assurance. 


Section 2. 4.6.4: Reactive Power 

Reactive power is that component of total power that is needed to maintain voltage and permit 
active power to be delivered. It is commonly measured in volt-ampere reactive (VAR) for small 
units and mega-VARs (MVAR) for large units. The requirement for reactive power to maintain 
grid integrity has become more important as electricity is transported greater distances because 
reactive power is consumed locally and cannot be transported long distances. The increased 
number of independent power producers has also contributed to this concern. On August 14, 
several entities were requesting reactive power support. The specific details involving reactive 
power on August 14 are discussed in Section 3.3. In our opinion, RTOs should take the lead on 
ensuring that there is adequate reactive power support on their systems. This need can be 
addressed by requiring all power producers to provide reactive power support in sufficient 
amount and requiring transmission owners to consider adding more capacitors to the system in 
their planning process. 

Reactive power issues are receiving more attention as a result of the blackout. On October 15, 
2003, NERC sent a letter to ail control areas and reliability coordinators. NERC requests that 
several near-term actions be taken, including seven involving the management of voltage and 
reactive power. These actions include: (1) establishing daily voltage/reactive management 


9 Reactive power issues overlap transmission and generation. Reactive power is necessary to maintain voltage on 
the transmission grid, but it is provided by local generation. This is the reason that the details are discussed in this 
report under utility operations rather than transmission. 
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plans, (2) ensuring reactive power supplies are verified and available, (3) having sufficient 
reactive power reserves, (4) maintaining voltage schedules, (5) reporting low voltage conditions, 
(6) ensuring generators have automatic voltage regulation, (7) coordinating potential differences 
of voltage criteria and schedules between systems. We believe that these are appropriate first 
steps to addressing the reactive power issues highlighted by the blackout. 


Section 2.4.6.S: RTO Authority 

The authority of the RTO should be clear and enforceable. Overall, the RTO should have the 
authority to enforce directives that are made to preserve grid integrity, especially during a time of 
emergency. The expansion of the RTOs seems to have led to a situation in the Midwest where 
the RTOs vie for transmission owners. The agreement that the transmission owner signs with the 
RTO must be geared toward ensuring the reliability of the system. PJM noted that its contracts 
with its market participants give it additional ability to oversee the system. As the newer RTOs 
move toward establishment of a market, they must also make sure that their contracts with 
market participants have a strong reliability component. One of the advantages that is provided 
by the RTO process is independence. The RTO must have no direct financial incentive to favor 
commercial market transactions that operate to the detriment of reliability. 

Control over reliability coordination functions should be consolidated within an RTO. The 
capability to respond to reliability emergencies must be quick and decisive. Decisions to save 
the grid cannot be subjected to lengthy negotiations among dispersed agents with potentially 
conflicting interests. Responsibility to act must be clearly established and the authority to carry 
out those actions firmly vested in the responsible party. As an emergency approaches, 
command and control must replace coordination as the decision-making modus operandi. The 
decentralized system now in place within MISO must give way to consolidation. Operating 
through 35 separate local reliability control areas, MISO’s capability to effectively perform its 
regional reliability responsibility is seriously compromised. Consistency and clarity regarding 
how reliability responsibilities are shared among the numerous independent control centers is 
problematic. Consolidation is desperately needed and we recommend centralization with all 
reliability responsibility placed under the direct control of MISO. While reducing the number of 
control areas would help, centralization is by far the more desirable option. Many of the other 
RTOs and ISOs in the Midwest, Northeast and Mid-Atlantic regions (PJM, New York ISO and 
the Ontario IMO) operate as centralized reliability control systems. Similarly, we recommend 
that AEP, currently operating as a satellite within the PJM RTO, be fully integrated and placed 
under the direct control of an RTO. 


Section 2.4.6.6: Transmission Seams 

The current RTO configuration resulting from the RTO choices of utilities in the_Midwest 
presents major transmission seams 10 problems. As utilities make and remake their choices based 
on a number of factors, RTO seams come and go along with those changes. This constant 
shuffling and reshuffling must stop. The fact is that utilities are connected. A seamless region 


10 Seams refers to the lines dividing the existing RTOs, where power passes from the control of one RTO to another. 
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will result in better reliability management, because the region would be overseen by one entity. 
This entity could then make choices based on the entire region, rather than pieces of it. The first 
step must be getting all of the Midwest utilities into an RTO. The shifting must stop and 
commitments must be made to stabilize RTO configuration in the Midwest After that, the RTOs 
that cover the Midwest can finalize a meaningful joint agreement so that the Midwest region is 
operated as a seamless area. 


Section 2.4.7: Demand Response 

Demand response, including distributed generation, must be effectively interwoven into a 
reliability improvement strategy. Historically, the demand side of the reliability picture has been 
largely overlooked in favor of the supply side of the equation. Demand response offers great 
potential to enhance reliability in a cost-effective manner. This opportunity must be aggressively 
tapped. System load balancing can be accomplished by adjusting either supply or demand. Both 
must be part of any cost-effective approach. Demand response should include pursuing load 
control opportunities, such as water heater control and air conditioning cycling equipment, which 
could greatly improve load-shedding capability. Demand response should also include 
distributed generation, which offers great value by locating smaller generation in strategic places 
on the grid that are vulnerable to reliability concerns. 


Section 2.5: Conclusions and Recommendations 

Thorough examination of the blackout event provides significant insight into reliability problems 
and concerns evident on August 14 ,h that significantly contributed to triggering the event along 
with actions that may have resulted in failure to localize the outage and prevent it from cascading 
over such a widespread area. Our findings include the following: 

• There is no evidence from our investigation to suggest that Michigan utilities or 
transmission operators were the cause of the blackout. All of the transmission line and 
power plant outages in the two and one-half hours before the two power surges beginning 
at 4:09 p.m. involved the facilities of FirstEnergy and AEP in Ohio. At the time that the 
power surges began, the electric system in Michigan was in balance. 

• MISO, as the regional reliability coordinator, should have informed affected transmission 
operators of the disturbances that were occurring in northern Ohio. 

• Michigan utilities and transmission companies were not notified of the problems being 
experienced by FirstEnergy and AEP and received no advance warning of the potential 
blackout. The first indication in Michigan of an impending emergency occurred at 
4:09:27 p.m. when an interconnection in central Michigan exceeded its emergency rating 
as a result of the first power surge coursing through the State. 

• Failure to isolate the FirstEnergy system from neighboring systems permitted the 
blackout to spread. The key to preventing blackouts from cascading is to take quick and 
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decisive action to localize them. The absence of effective support to contain disturbances 
from regional coordinators responsible for reliability within the Ohio area where the 
critical grid disturbances occurred is disturbing. Reliability coordinators responsible 
include MISO on behalf of FirstEnergy and PJM for AEP. Representatives from all four 
organizations were involved in discussions regarding the disturbances, yet no one entity 
was able to see the whole picture and put the pieces of the puzzle together. Among the 
most obvious improvements that need immediate attention is clarification of reliability 
responsibilities and how they should be executed. Control room communications and 
statements by industry grid managers and reliability coordinators in our investigation 
clearly reveal confusion on this most critical Junction. Enforcement responsibility, 
authority, and accountability must be clearly defined and strengthened. Structural 
changes in how MISO executes its coordination duties must be revised. 

• FERC should be provided with the authority to develop and enforce reliability standards. 
Reliability standards enforcement is inadequate and in urgent need of significant revision. 
The current system to enforce reliability standards lacks accountability and is generally 
ineffective. The present enforcement structure relies exclusively upon voluntary 
standards with no governmental agency responsible for oversight. Reliability standards 
must become the centerpiece of a comprehensive strategy to improve grid reliability. 
Reliability standards should be: (1) nationally developed and applied, (2) mandatory, (3) 
strictly enforced by the FERC, and (4) implemented regionally through the RTOs. FERC 
should have the responsibility to develop and oversee this process and that agency should 
be held accountable for the results. 

• A single RTO should be established for the Midwest region. Regional grid coordination 
and management through RTOs in the Midwest is not strong enough and seams between 
RTOs are causing serious problems that must be addressed. Preventing blackouts from 
recurring requires a strong and effective regional response. In the Midwest, RTO 
participation must by mandatory. Cascading outages are an unfortunate byproduct of 
integrated systems; solutions designed to prevent them require a coordinated response. 
Coordination is not an option if the nation is committed to seriously addressing 
reliability; it is a must. 

• Consideration should be given to building additional transmission in the region. 
Curtailments of firm transmission have increased each of the last four years. 

• The MISO reliability coordination structure is flawed. Reliability coordination within the 
MISO system is highly decentralized, with 23 independent transmission companies 
operating as their own control area operators. MISO, under this fragmented structure, in 
practice, operates as reliability coordination back-up, with front-line responsibility 
dispersed among the local reliability control operators. This system is fraught with 
inconsistency and confusion as to delineation of responsibility to monitor the grid and 
execute reliability control measures. MISO reliability coordination operates like a loose 
confederation, significantly undermining MISO’s ability to act promptly and decisively. 
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• Electric grid reliability is in need of support on several levels. There is no quick fix to 
prevent future blackouts. The solution is complex and must be comprehensive. It 
involves elements that are interdependent and will encompass both short and long-term 
remedies. Key to a successful response are four interdependent components which must 
be substantially expanded and strengthened: 

1) Reliability standards; 

2) Regional grid coordination and management; 

3) Strategic grid investment, where necessary, to cost-effectively expand and 
upgrade the transmission infrastructure; and 

4) Demand response, including distributed generation. 

To become most effective, these grid improvements should be implemented as a 
comprehensive package. It would be a mistake to pursue a piecemeal approach. 


34 



411 


PART III 

UTILITY OPERATIONS 


Section 3.1: Michigan Electric Utility Systems 
Section 3.1.1: Consumers Energy Company 

Consumers Energy Company provides electric service to more than 1 .7 million customers and 
serves 275 cities and villages in 61 counties. Principal cities served are Battle Creek, Bay City, 
Cadillac, Flint, Grand Rapids, Jackson, Kalamazoo, Midland, Muskegon and Saginaw. The 
company operates 12 coal-fired and two oil-fired generating plants, 13 hydroelectric plants, a 
pumped storage generating plant, and several combustion-turbine plants that produce electricity 
when needed during peak demand periods. The company owns the Palisades nuclear plant. The 
utility also purchases power from several sources, such as the gas-fired Midland Cogeneration 
Venture. Major generation facilities include: the J.H. Campbell Generating Complex, J.R. 
Whiting Plant, the D.E. Kam - J.C. Weadock Generating Complex, the B.C. Cobb Generating 
Plant, the Palisades Nuclear Plant, Ludington Pumped Storage Plant, and a number of small 
hydro units. 

The J.H. Campbell complex, located on the shore of Lake Michigan between Holland and Grand 
Haven, is Consumers Energy's largest coal-fired generating complex. Unit 1 began operation in 
1962, Unit 2 in 1967 and Unit 3 in 1980. Three turbine-generators produce up to 1,404 
megawatts (MW) of electricity. 

The Whiting Plant is located on the Lake Erie shoreline of southeastern Michigan. The Whiting 
Plant produces up to 3 1 0 MW of electricity, enough to power a community of 230,000 people. It 
was built in 1952. 

Situated at the mouth of Saginaw Bay, the Kam-Weadock complex generates about one-third of 
all the electricity produced by Consumers Energy. In 1940 Consumers Energy built the first of 
what would eventually become eight units on the J.C. Weadock site. Two of those units are still 
in operation. Between 1959 and 1977, two plants were added: D.E. Kam 1 and 2 and D.E. Kam 
3 and 4. The boilers in Weadock and Kam 1 and 2 bum coal; at Kam 3 and 4, they bum natural 
gas and oil. At peak operation, the plants can produce 2,100 MW of power. 

The B.C. Cobb Generating Plant is located on the shores of Muskegon Lake, where its water 
meets the Muskegon River. The plant's five coal and natural gas units can generate 500 
megawatts of electricity. 

The Palisades plant, located near South Haven, has been generating electricity since 1971, and 
represents about 18 percent of Consumers Energy's total electrical capacity. In November 2000, 
Consumers Energy signed an agreement to become a full partner in Nuclear Management 
Company (NMC) of Hudson, Wisconsin. As part of the agreement. Consumers Energy 
transferred responsibility for the operation of Palisades to NMC. Consumers Energy retains 
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ownership of Palisades, the electricity it produces and its spent fuel. The utility also retains the 
financial obligations for the safe operation, maintenance and decommissioning of the plant. 

The Ludington Pumped Storage plant located on Lake Michigan near Ludington can generate 
1872 MW of electricity. Customers throughout Michigan use energy generated by the facility. 
Because its six turbines can begin generating within a few minutes, the Ludington plant can 
respond quickly to daily, weekly and seasonal changes in energy demands. The Ludington plant 
operates very simply. At night, when demand is low, the facility's six reversible turbines pump 
water 363 feet uphill from Lake Michigan. The water is pumped through six large pipes, or 
"penstocks," to an 842-acre reservoir. During the day, when demand is high, the reservoir 
releases water to flow downhill through the penstocks. The flowing water turns turbines in the 
powerhouse to make electricity. 

Consumers Energy purchases from small hydroelectric facilities located through out the state that 
provide 1 14.9 MW of power. Consumers Energy also owns 345 MW of combustion turbine 
peakers located throughout the state. The largest group of peakers is at the Thetford facility, 
which has 192 MW of peakers. There are also 13 MW located at the Campbell facility, 70 MW 
at Gaylord, 28 MW at Morrow, 16 MW at the Straits, 13 MW at the Weadock facility and 13 
MW at the Whiting facility. 

Power transmission for Consumers Energy’s system is provided by Michigan Electric 
Transmission Company (METC) - a privately held company. METC acquired Consumers 
Energy’s transmission assets in 2001 and is currently operated out of the Jackson dispatch center. 
The METC system is interconnected with the American Electric Power system near that utility’s 
Donald C. Cook Nuclear Plant near Bridgman. The transfer capability of this interconnection is 
rated in 4,000 to 4,500 MW range, depending on the actual configuration of the power system. 1 11 


Section 3.1.2: The Detroit Edison Company 

The Detroit Edison Company, the largest electric utility in the State, generates and distributes 
electricity to 2.1 million customers in a 7,600 square-mile service territory in Southeastern 
Michigan. The utility operates 10 base-load generating plants, all within its service area. The 
company also is co-owner with Consumers Energy of the Ludington Pumped Storage facility. 
Detroit Edison’s system capacity totals nearly 1 1 ,000 MW. Coal is used to generate about 85 
percent of its total electrical output, with the remainder produced from nuclear fuel, natural gas 
and solar energy. 

The Fermi 2 nuclear plant is located in Monroe County, south of Detroit on the west end of Lake 
Erie. Fermi is a boiling water reactor with an in-service date of January 1998 and a summer 
capability of 1 ,1 1 1 MW. 

Placed in service in July of 1 979, Greenwood is a conventional steam turbine unit that was 
designed to bum residual oil. It has been converted to dual fuel burners, with the capability to 


1 ' These numbers are circuit ratings and may not reflect the actual transfer capability of the interconnection at any 

given time. Actual transfer capabilities could be less. 


36 



413 


also bum natural gas. Located at the Greenwood energy center in St. Clair County it has a 
summer rating of 785 MW. 

The Monroe Power Plant is a 3,000 MW facility located south of Detroit in Monroe County. It 
consists of four coal-fired units, each with a summer capacity rating of 750 MW. Unit 1 went 
into service June 1971, followed by unit 4 in March 1973, unit 2 in May 1973, and unit 3 in May 
1974. 

The St. Clair facility, located in county and along the river of the same name, has six active coal- 
fired units. St. Clair 7 (placed in service in 1969) is the largest with a summer capability of 45 1 
MW, followed by unit 6 (placed in service 1961) with a summer capacity of 321. Units 1 
(placed in service in 1953) and 4 (placed in service in 1954) are rated at 158 MW each, while 
unit 3 (placed in service 1954) is rated at 168 MW, and unit 2 (placed in service in 1953) is rated 
at 162 MW. 

The Belle River Power Plant consists of two coal-fired units, each with capability of 635 MW. 
Unit 1 was placed in service in August 1984 and unit 2 in July 1985. The plant, with a total 
capability of 1 270 MW, is located near the St. Clair facility. 

The Trenton Channel facility located in Wayne County on the Detroit River just north of Detroit 
has 3 active units. Unit 9, which is coal-fired, is the largest with a summer rating of 535 MW. 
Units 7 and 8 are each 120 MW coal-fired units. 

The River Rouge Facility, located within the city of Detroit, has two active coal-fired units 
owned by Detroit Edison. Unit 3, placed in service in 1958, has a summer rating of 276 MW. 
Unit 2, placed in service in 1957, has a rating of 238 MW. Unit 1, placed in service in 1956, is 
rated at 199 MW. It has been converted to natural gas and transferred to DTE Energy, the parent 
company of Detroit Edison. 

The Harbor Beach plant is located in Huron County near the town of Harbor Beach on Lake 
Huron. Placed into operation in 1968, the single 103 MW unit bums coal. 

Detroit Edison has a total of 1,371 MW of peaking units 12 with 600 MW located at base load 
generation facilities and 771 located at various other locations throughout the utility's system. 
Other significant generation 13 owned by companies other than Detroit Edison in its service 
territory include: CMS Energy’s Dearborn Industrial Generation rated at 330 MW, First 
Energy's Sumpter-Dayton Station rated at 300 MW and DTE Energy Services Dean Plant rated 
at 300 MW. 


12 Base load and peaking units are distinguished by cost and operating characteristics. Base load units generally are 
expensive to build but have low operating costs - they are expected to run a high percentage of the time to take 
advantage of the low variable cost and spread the high fixed cost over a larger volume of output. Conversely, 
peaking units are relatively cheap to build but have high operating costs - they are expected to operate only a small 
percentage of the time (10% or less). 

13 A map of merchant plants in Michigan is available in PDF format from the State Utility Forecasting Group at 
Purdue University. See https://enaineering.ourdue.edu/IIES/SUFG/MAPS/index html . 
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International Transmission Company (ITC) owns and operates the transmission lines associated 
with the Detroit Edison system. It is made up of transmission facilities formerly originally 
owned by Detroit Edison, which were sold to an independent transmission company in 2002. 
Detroit Edison currently switches, repairs and maintains the ITC transmission assets under 
contract. This arrangement will continue for one year. Switching is controlled from Detroit 
Edison’s Systems Operation Center. 

Most of the control area operation in Michigan is done by the Michigan Electric Coordinated 
Systems (MECS) at the old Ann Arbor Power Pool. MECS is comprised of ITC and METC, the 
two independent companies that operate the Michigan grid. 

The Detroit Edison/ITC system has 345 kilo-Volt (kV) interconnection with FirstEnergy at Allen 
Junction-Majestic-Monroe, Lemoyne-Majestic and Bayshore-Monroe. All of the 
interconnections are in the southeast comer of Michigan. The Detroit Edison/ITC system is 
interconnected with Hydro One in Ontario at the 345 kV level through two interconnections at 
Lambton-St. Clair near Detroit Edison’s St. Clair power plant. The total transfer capability with 
FirstEnergy is 3,380 MW, and 2,400 MW with Hydro One. 

The Consumers Energy/METC system has four 345 kV interconnects with the Detroit 
Edison/ITC system: (1) Majestic-Battle Creek-Onedia, (2) Majestic-Tompkins, (3) Thetford- 
Jewell, and (4) Pontiac-Hampton. These interconnections have a combined continuous rating of 
3,000-3,500 MW depending on the power flow and a short term rating in the 4,000-4,500 MW 
range. 


Section 3.1.3: American Electric Power 

American Electric Power (AEP) serves customers in the lower western portion of the state. Its 
major generation facility within Michigan is the Donald C. Cook Nuclear Power Plant, located 
on the shores of Lake Michigan near Bridgman. The 1 ,020 MW Unit 1 went into commercial 
operation in 1975, while the 1,090 MW Unit 2 was completed in 1978. AEP’s 765 kV 
transmission system is interconnected to the Consumers Energy/METC system near the 
Donald C. Cook plant. 


Section 3.1.4: Lansing Board of Water and Light 

The Lansing Board of Water and Light (BWL) is a municipal utility, owned by the citizens of 
Lansing. The BWL is the third largest electric utility in the state serving 98,000 customers and 
the largest municipally-owned utility in Michigan. The BWL has two generating stations: 
Erickson (located west of Lansing) and Eckert (located near the center of the city). The Erickson 
Station was completed in 1973 and contains a single coal-fired unit capable of producing 159 
MW. It was recently rated one of the most efficient plants of its size in the United States. 

Eckert Station Located near downtown Lansing was constructed in the early 1920's and has 
undergone several rebuilding and expansion projects. The Eckert Station includes six electric 
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generating units ranging from 41 to 77 MW. The six units are capable of generating a total of 
351 MW. 

Through a membership in the Michigan Public Power Agency (MPPA), the BWL also receives 
146 MW of electricity from the Belle River Plant operated by Detroit Edison. The BWL is 
interconnected with Consumers Energy in two locations. 


Section 3.2: Emergency Procedures 
Section 3.2.1: Load Management 

Three load management mechanisms are available to interrupt electric load when generation 
supply, system voltage, or system frequency become deficient: (1) Rotating Load Management 
(RLM), (2) Remote Load Shed (RLS), and (3) Automatic System Security. 

On the Consumers Energy system the RLM approach utilizes rotating blackouts involving 
Supervisory Control and Data Acquisition (SCAD A) control of substation circuit breakers. 
Detroit Edison uses a manual system to accomplish the same function. RLM is designed to 
interrupt firm load in the event of a generation shortage or when a widespread transmission 
system emergency exists. The RLM system is intended to be implemented prior to initiation of 
Automatic System Security. 

For example, on the Consumers Energy system there are fourteen RLM load segments, each with 
an average load of 89 MW. Twelve of the segments can be implemented via use of SCADA 
control of substation circuit breakers. Each segment consists of an average 110 MW of load 
from each of three System Control Center areas - South, West and East. The two remaining 
segments are manual industrial segments that require operator intervention at the 
substation/customer site to interrupt load. The Detroit Edison system operates in a similar 
manner, except the system uses a manual control process. 

No interruptions of hospitals with surgical facilities are included in the plan. 

Depending upon the electric system requirements, any segment or part of a segment can be 
interrupted by opening selected 23 kV, 46 kV, and 138 kV circuit breakers via SCADA (on the 
Consumers Energy system), certain customer substation equipment or customer switchgear 
equipment. Designated customer substations, sub-transmission and transmission lines will be 
interrupted for two hours. Customers would be interrupted for two to four hours (or longer if 
equipment failure or system malfunction occurs). If required, the next RLM segment will be 
interrupted. Restoration of the prior RLM segment will commence via SCADA on most of 
Consumers Energy’s system and manual intervention Detroit Edison’s system. 

Consumers Energy and Detroit Edison have the capability to implement RLS to drop load during 
isolated transmission system problems, which do not require system-wide operating intervention 
or RLM capabilities, implementation of RLS is normally the first step after an Emergency 
Condition has been declared. 
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On the Consumers Energy system, thirty-three toad segments, ranging from 30 to 271 MW, have 
been identified for RLS purposes via SCADA control of substation circuit breakers. The priority 
of the RLS segments is based on the nature and location of the transmission system emergency. 
Depending upon the electric system requirements, any segment can be dropped by opening 23 
kV and 46kV circuit breakers via SCADA. Designated sub-transmission lines will be dropped 
and remain de-energized until the transmission system deficiency has been mitigated. 

Both Consumers Energy and Detroit Edison have Automatic System Security under frequency 
relays installed on selected transmission substation circuit breakers and distribution substation 
circuit reclosers. These relays will trip and block reclosing of the circuit breakers and reclosers 
if the system frequency declines below preset values. The intent of this system is to prevent 
widespread electric system disruption and ultimately system-wide blackout should manual 
intervention (RLM or RLS) not be implemented quickly enough to stop frequency decay. 

On the Detroit Edison system, automatic load shedding did operate as designed at the beginning 
of the blackout. During the event of August 14, no manual load shedding was implemented due 
to the absence of advance warning. On August 14, prior to the blackout, both systems were in 
normal operating condition with all safety devices in service and functional. 


Section 3.2.2: System Protection 

The transmission and distribution systems are considered to be stable when voltage, frequency 
and thermal loading are within normal operating ranges without dramatic variances. In 
Michigan, the system is designed to withstand the loss of one or more key components. 

Key transmission lines and distribution circuits are monitored for excessive current. If an 
overload condition is detected, the protective relaying or other equipment will interrupt the 
circuit to prevent damage. The main purpose of this electrical protective equipment is to protect 
the physical components and maintain electrical integrity of a power system against faults (short 
circuits). 

System frequency is usually maintained at 60 Hertz, 14 but power shortage or oversupply 
conditions can affect this frequency. For example, if there is an undersupply of power, 
frequency will drop. This can cause damage to customer or utility equipment. Electric utilities 
maintain protective relaying systems to monitor for these conditions and to isolate the source. 

When reactive power is under or over supplied, voltage will decrease or increase. Either of these 
conditions can damage critical equipment owned by the utility or the customer. Once again, 
protective relaying is required to prevent damage. 

Protection schemes are designed to rapidly isolate a failed or faulted component or segment of 
the power system to minimize both its effect on the rest of the power system and damage to the 
affected component. This allows the remainder of the system to continue to operate normally. 


14 Hertz is a unit of frequency equal to one cycle per second. 
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In some cases, lightning strikes for example, faulted line segments are automatically restored to 
service. 

The level of sophistication, redundancy and type of protection equipment used depends on the 
voltage level of the system on which it is installed. Most protective relays monitor current, 
voltage, and combinations of these, to determine if an abnormal condition exists. The protection 
engineer sets the relays to respond to the conditions predicted in system models. Relay schemes 
are generally divided into phase protection and ground protection. The phase schemes detect 
when conductors for different phases contact each other. Ground protection schemes detect 
faults to earth ground, such as lightning. These relays schemes are typically located in the 
substations and trip the appropriate circuit breakers at line terminations. 

On 345 kV transmission systems, most protective schemes are communication based. This 
means that the protective relays at each end of the line “communicate” with each other via power 
line carrier, audio tone (phone lines), or fiber optics, in order to determine if a fault exists. This 
provides complete coverage for high speed clearing of the entire line. METC and ITC systems 
typically have three levels of redundancy: two primary systems that rely on relay 
communication and a backup system that operates independently at each end. 

On 138 kV transmission systems, some communication-based protection exists where it is 
required for grid stability or relay coordination. In general, the remaining relay protection is 
impedance-based, effectively measuring an electrical “distance” down the line. If a fault lowers 
the impedance to a value that falls within its zone of protection, the relay operates. Most 
schemes have three zones of protection, operating at different current levels and time delays in 
order to assure that only the faulted segment is isolated. Zones overlap to provide varying 
degrees of redundancy. 

On the 46 kV high-voltage distribution systems, the relay protection is less sophisticated because 
less protection is required. Most of the phase fault protection consists of coordinated time-over- 
current elements that are controlled by impedance-based elements that determine directionality 
of the fault current. If current flow indicates that a fault could be located downstream, the relay 
operates. This operation occurs after a specified time delay based on the magnitude of the 
current flow. The time delay allows the protective devices electrically closest to the faulted 
equipment to operate first. 

The North American Electric Reliability Council (NERC) states that a security coordinator must 
ensure the integration of reliability practices within an interconnection and market interface 
practices among regions. The security coordinator is responsible for recognizing alert conditions 
and providing notification to control areas and transmission providers. Alert conditions include 
cases where energy requirements cannot be met or resources cannot be scheduled. Transmission 
Load Relief (TLR) is a mechanism for a security coordinator to curtail or re-dispatch scheduled 
transactions to keep the use of the grid within its operating limits. The security coordinator for 
the Consumers Energy/METC system and the Detroit Edison/ITC system is the Midwest 
Independent System Operator (MISO). 


41 



418 


Section 3.2.3: Individual Generating Units 

Individual generators are protected from damage by protective relays. These relay systems sense 
conditions 15 and isolate the generator from the system if any of the conditions exist or are out of 
range. Individual relays are on each unit. Also, additional multifunction digital relays are on many 
units. The multifunction relays have several protective functions incorporated into one device, soms 
of these protective functions duplicate the features of individual relays, but others provide alarms 
only. 

When a generating unit is in the process of being shutdown during a planned outage, steam flow 
from the boiler to the turbine is decreased through control valves to gradually reduce the 
electrical load on the unit. This is coordinated with other generation and transmission 
organizations to ensure system stability. Once the unit’s load has been reduced, the main unit 
breaker is opened. Remaining operating plants increase outputs slightly to satisfy the real and 
reactive load being served by the unit being shutdown. Numerous procedures are then 
performed on the off-line unit to prevent damage from residual thermal effects or other physical 
conditions. This gradual unloading process cannot be followed during an emergency shutdown. 

In such a case, the main unit breaker trips with the control valves between the boiler and turbine 
closing. The dramatically increased steam pressure in the turbine and boiler are relieved through 
equipment and procedures designed to protect personnel and equipment. Returning a plant to 
service after an emergency shutdown requires time-consuming efforts due to the possibility of 
damage resulting from the rapid shut down. 

Under normal operation, the pressure of the steam generated in boilers in power plants is 
proportional to the load the plants are serving. When electrical load is suddenly removed from 
the electrical generator, the steam valves, which feed steam from the boiler to the turbine, close 
to bring the plant to a rapid but controlled shutdown. During this process, the pressure in the 
low-pressure turbine may increase suddenly. Rupture discs are designed to relieve this pressure. 
They are incorporated in power plants in the low-pressure section of the steam turbine to prevent 
injury to personnel or damage to equipment. If the rupture discs do not perform this function, 
the turbine generator could be damaged due to an over-speed condition, or the steam turbine 
housing could rupture. Either of these situations could injure workers and require months or 
years to repair. Once a rupture disc has operated, it must be replaced before a unit can be 
returned to service. 

When a power plant shuts down, residual thermal effects can distort the Machine Turbine Generator 
(MTG) shaft or damage its bearings. A sequence of operations must be carefully followed to 
prevent any of these critical elements of the system from being damaged. Repair of these elements 
can take months or years. For example, the MTG shaft must continue to turn after steam ceases to 
enter the turbine - if this motion does not continue, the hot shaft could warp under its own weight. 


15 Examples include: phase differential, ioss-of-field, negative sequence, volts per hertz, stator ground over-current, 
reverse power, under-frequency, inadvertent energization, stator ground over-voltage, phase distance, generator field 
ground and generator field over-excitation 
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Section 3.3: The Blackout 

The events leading up to the blackout are discussed in Part I. The initial impact appeared on the 
Michigan system at 4:09 p.m. on August 14, 2003, as a large load in northern Ohio. This load 
was initially balanced by power coming into Michigan on the METC system through its 
interconnection with the AEP system in southwestern Michigan and flowing through the ITC 
system to FirstEnergy. As described in Section 1.3.1, a rapid series of cascading failures caused 
violent power surges to flow through Michigan, ultimately causing the interconnects to trip 
between Consumers Energy, Detroit Edison and surrounding utilities. Some of these 
interconnections flows are shown in Charts 3.1 and 3.2. 

Chart 3.1 
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Chart 3.2 
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Both Consumers Energy and Detroit Edison report they had no knowledge of the problem 
developing between FirstEnergy and AEP. In addition, they had no advance knowledge 
regarding the impending event. The short lead-time (seconds rather minutes) prior to this event 
on the Michigan system did not allow for human intervention by any entity connected to the 
system. All actions taken were initiated by automatic switching equipment under the direction of 
the logic embedded in the systems that control interconnection relays. The protective equipment 
viewed the increase in load and the general voltage collapse as a fault to ground. Michigan 
power providers received no advance warning to enable them to protect their customers from the 
outage. 

During the initial seconds of the event, as experienced in Michigan, the Consumers 
Energy/METC system, which had been transferring about 2,000 MW of power into the Detroit 
Edison/ITC area prior to the event, separated from the Detroit Edison/ITC system under the 
additional load placed on it by the FirstEnergy load in Ohio. The load across the east-west 
interconnections exceeded 4,200 MW on an interconnection with a short term rating of about 
4,000-4,500 MW and a 24 hour rating of 3,000-3,500 MW. See Chart 3.1. During this same 
period the system was experiencing a general collapse in voltage - for example, the voltage at 
the ITC Bayshore 345 kV interconnection with FirstEnergy dropped to below 305 kV and 
Detroit Edison ’s voltage at its Pontiac Station dropped to 78% of the normal voltage. The 
protective equipment was designed to interpret such a voltage collapse as a fault to ground. 
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During the initial phase of the event Consumers Energy’s Campbell unit #3, the MCV power 
facility, the Jackson-based Kinder Morgan facility and most of Detroit Edison’s major units were 
tripped offline by their automatic protection systems. See Charts 3.3, 3.5, and 3.6 for details. In 
some cases the reactive load requirements placed on generation units exceeded the maximum 
capabilities by over 300%. 

The pre-event loadings coupled with the increased requirements to support FirstEnergy through 
ITC simply overloaded the Michigan systems and caused the interconnections to open isolating 
the Detroit Edison system from the Consumers Energy generation and the Ludington Pumped 
Storage Facility. Consumers Energy was able to continue serving most of its customers through: 
(1) its own generating units, which, except for Campbell # 3 and the three Whiting Units, stayed 
on line; (2) AEP power, including power that had been flowing through METC to ITC/Detroit 
Edison; and (3) Detroit Edison’s share of the Ludington Pumped Storage Facility. Some 
Consumers Energy load was lost in the Flint area during the disconnection of METC from ITC 
and several southern counties lost power due to the outage of the Whiting Power Plant. 

Chart 3.3 
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The August 14 event heavily impacted the Michigan electric power systems from a reactive 
power standpoint. Reactive power must be present at sufficient levels to maintain voltage. 
Without adequate reactive power support, voltage levels fall and the system becomes unstable. 
The reactive power component cannot be transported long distances as can the active power 
component and must be produced fairly close to the actual load either by spinning generators or 
capacitor banks. 
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At 3:30 p.m. on August 14 (prior to the blackout), the Michigan generation was supplying about 
300 megavars (MVAR) to FirstEnergy over the ITC/FirstEnergy interconnection. The MVAR 
required to support FirstEnergy increased slowly over the 30 minute period preceding the event 
until it reached just over 400 MVAR at 4:06 p.m. At that time, it jumped to nearly 700 MVAR, 
where it remained until the METC and the ITC systems separated at 4: 1 0 p.m. After the 
separation the MVAR flow into FirstEnergy from ITC was near zero. 

During this period, the METC system was supplying roughly 200 MVAR into the AEP system in 
support of the 2,000 MW of power that was being transferred into Michigan over the 
METC/ AEP interconnection. The Consumers Energy/METC system was also supplying about 
200 MVAR into the Detroit Edison/ITC system just prior to the event. As the event unfolded, 
the MVAR transfer into the Detroit Edison/ITC system increased to nearly 300 MVAR at about 
4:10 p.m., then reversed and became a 700 MVAR draw from the DE/ITC system just as the 
event occurred. 

Michigan based generating units experienced an extremely large MVAR draw just prior to the 
event and during the initial phase of the event. Consumers Energy reports that all of the 
connected generation experienced MVAR requirements beyond their normal capabilities. 
Generating units are not capable of supporting MVAR requirements at elevated levels for a 
lengthy time without sustaining serious damage. Chart 3.4 displays the MVAR variation 
experienced by CE’s Campbell unit 3 during the event. 


Chart 3.4 
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Chart 3.5 


Consumers Energy Generation Status 
Before and After the Event 
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The large MVAR draw on the Michigan system signifies a voltage collapse somewhere on the 
interconnected system. A further complication of the situation occurs as the MW draw across 
the Michigan system jumped to over 3,000 MW and subsequently above 4,200 MW. This power 
transfer across Michigan used up a large portion of the available MVAR support to support the 
transfer. Unable to maintain the required MVAR support the Michigan systems entered a 
voltage collapse situation that precipitated the tripping of the generation that Detroit Edison had 
on-line. With the interconnection with Consumers Energy/METC system open and with almost 
all of its generation tripped off-line, Detroit Edison was unable to support its internal load. 
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Chart 3.6 


Detroit Edison Generation Status 
Before and After the Event 
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Section 3.4: Review and Analysis of Actions Taken 

The events that occurred outside of the State that preceded the outage were beyond the view of 
the power system operators within Michigan. This and the lack of advance warning from MISO, 
First Energy, AEP, or any other organization, prevented the Michigan electric system operators 
from taking manual action to protect customers from the impending outage. Even if the 
Michigan electric system operators had full knowledge of the generation and transmission line 
outages unfolding between FirstEnergy and AEP, actions open to them might have presented a 
significant risk to the stability of the Michigan system. Isolating the Michigan system from its 
interconnections with outside utilities would have been difficult at best and the results uncertain 
because Detroit Edison was importing about 2,900 MW through the METC/ITC system from the 
METC interconnect with AEP. 
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If Michigan operators had full knowledge of the events occurring elsewhere, it appears that 
approachs other than Ml isolation may have been available to prevent or at least significantly 
reduce the pending power surges. In order to prevent the power surge from coming through 
Michigan, the ITC/FirstEnergy interconnect could have been opened prior to the event. If it is 
not done before the event, once the interconnect is heavily loaded, opening it would likely have 
forced Detroit Edison generating units off-line due to turbine over-speed from the sudden loss of 
load. This could have caused a power outage on the Detroit Edison system. In any case, the lack 
of information from other systems prevented Michigan operators from exercising or considering 
this option. 

The other course of action that may have prevented the outage in the Detroit Edison service area 
would have involved unloading the interconnections between the Consumers Energy/METC 
system and the Detroit Edison/ITC system in advance of the event. An unloaded interconnection 
may have weathered the coming storm causing the interconnections between ITC and 
FirstEnergy to open sparing the Detroit Edison system from a full blackout. If Detroit Edison or 
ITC had received advance notice of the problems experienced elsewhere, they may have been 
able to increase generation levels within the Detroit Edison territory, thereby unloading the ties 
with the Consumers Energy/METC system. If Michigan companies had been provided enough 
advance notice of the events 16 occurring between FirstEnergy and AEP, such action may have 
reduced or eliminated the impact on Detroit Edison and Consumers Energy customers. 

Detroit Edison generation on-line at the time of the blackout had the capability to provide an 
additional 1,600 MW of power. If Detroit Edison had known of the problems being experienced 
elsewhere, it could have ramped up its internal generation prior to the event to full power levels 
in the time period between the FirstEnergy Hanna-Juniper 345 kV trip at 3:32 p.m. (this was the 
second 345 kV line to trip within the FirstEnergy system, the first was Chamberlain-Harding at 
3:06 p.m. EDT) and 4:09 p.m. This would have reduced the transfer across its interconnection 
with METC by 1,600 MW to approximately 1,300 MW. With the transfer level in this range a 
Detroit Edison system blackout could probably have been avoided. 

Detroit Edison could not exercise this option because the key information regarding the 
impending system disturbance was not shared beyond FirstEnergy, AEP and MISO. Failure to 
share key reliability information with Detroit Edison or ITC also prevented ITC from 
considering the option of opening its interconnections with FirstEnergy, or at least opening two 
of the three to allow automatic interruption of the remaining connection prior to the opening of 
the METC/ITC interconnections. 

It appears based on the results of this investigation that the protection systems located within 
Michigan worked as designed. The Detroit Edison/ITC system had too much power flowing into 
its system to survive a separation from the Consumers Energy/METC system without major 
generating units tripping off-line. The Consumers Energy/METC system was able to separate a 
major portion of its system from the problem. The fact that there was no major damage either to 
transmission and associated equipment or to generating units and their associated equipment 
demonstrates that the automatic protection equipment worked as designed. If this equipment had 


16 Four organizations, MISO, PJM, FirstEnergy, and AEP, had information about these events. Any of them could 
have, and should have, provided the information they had to other affected entities. 
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not worked as designed, there would likely have been major damage to facilities that could have 
taken weeks or even months to replace or repair. 


Section 3.5: Recovery 
Section 3.5.1: Consumers Energy 

Restoration efforts were undertaken immediately following the event. Consumers Energy issued 
an emergency page to notify departments within its Transmission & Distribution organization 
that a significant system disturbance had taken place and requested that leadership personnel 
report to the nearest System Control Center. Local headquarters in the affected areas were also 
instructed to remain open. A conference call was established at 5:15 p.m. to determine initial 
actions. Subsequent calls were held every two to three hours thereafter. Independent conference 
calls were also held with METC on a similar schedule. 

Personnel in the Merchant Operations Center assessed generation status and established 
communications with operations centers within Michigan and nearby systems. Management 
from Fuels & Power Transactions and Nuclear, Fossil and Hydro Operations organizations 
arrived by 4:30 p.m. to monitor the situation and direct restoration efforts from the supply side. 
Merchant Operations personnel opened communication with DTE merchant personnel as well as 
transmission operators to assess the extent of the disruption on supply resources. 

On the Consumers Energy/METC system there were significant generator outages, numerous 
line outages, and two areas were without power. Generation outages included Midland 
Cogeneration Venture (1,240 MW) in Midland, Kam 2 (260 MW) in Bay City, Campbell 3 (820 
MW) in Port Sheldon, and Whiting 1-3 (330 MW) in Monroe. Line outages included two 345 
kV ties and two 138 kV ties between METC and the Detroit Edison/ITC system, two 138 kV ties 
between METC and Lansing BWL, one 138 kV tie between METC and Northern Indiana Public 
Service Company (NIPSCo), and a multitude of 138 kV and 46 kV lines in the southeastern 
portion of the state. Additionally, there was one 46 kV line in the Flint area that was out of 
service and numerous distribution circuits locked out. The two major areas without power were 
the Lansing BWL and the southeast comer of Consumers Energy’s service territory 
(geographically enclosed predominately by 1-94 to the north and M-66 to the west). 

Immediately following the event, Consumers Energy started generation in response to the loss of 
units. Consumers Energy believed at that time it was under-generating, but interconnection 
frequency continued to be above 60 Hertz, which would generally be an indication of over- 
generation. In consultation with transmission operators. Consumers Energy maintained its 
generation level until the status of the system, both in Michigan and in neighboring areas, could 
be assessed. Between 5:00 p.m. and 7:00 p.m. power output from the Ludington Pumped 
Storage facility was reduced in order to moderate high frequency levels and manage available 
stored water for later restoration needs of Detroit Edison. Consumers Energy also obtained 
additional supplies of electricity from in-state independent power producers and AEP. 
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Restoration efforts followed black start 17 procedures. Efforts began with an assessment of the 
138 kV and 46 kV breakers that were open. The open breakers were plotted on a geographic 
map of the electric system in order to determine the boundaries of the affected areas. Having 
defined the affected area. System Control Centers began the process of opening up all breakers 
contained within the affected area via SCADA and field personnel. 

Consumers Energy’s Lead System Control established communication with the Electric Sourcing 
and Trading group to keep apprised of generation status. The Lead System Control also 
established communication with the Michigan Electric Power Coordination Center, ITC and 
MISO to keep apprised on issues pertaining to the transmission system and its interconnections 
with neighboring utilities. 

Consumers Energy’s merchant group maintained communication with its Detroit Edison 
counterpart, as well as other control centers. Consumers Energy agreed to fill Ludington 
Pumped Storage overnight for both itself and for Detroit Edison to prepare for restoration efforts 
the next day. 

The return of generation at the Whiting facility and the restarting of generators at Kinder Morgan 
power plant were a top priority. These units provide both local power supply and area voltage 
support. Also, Consumers Energy believed that the automatic relays (which are designed to 
isolate distribution circuits for sustained periods of under-frequency operation) had tripped and 
that personnel would need to be dispatched to reset these relays and energize the tripped circuits. 
Consumers Energy personnel from the western and central portion of Michigan were dispatched 
to the affected area to aid in resetting the relays. 

As of 5:15 p.m., all breakers within the affected area were opened, and the process of restoring 
the 138 kV system was undertaken. This included service to Whiting Substation, which 
provided station power for restarting Whiting Generation. The 138 kV system, with the 
exception of the ties with ITC and NIPSCo, was restored by 7:25 p.m. During restoration of the 
138 kV system some 46 kV and 138 kV connected load was also restored. At that time System 
Control began energizing the remaining 46 kV lines and restoring customers to service. At this 
point, the system was net generation deficient. Therefore, restoration of customers progressed at 
a pace relatively equal to the rate at which generation became available. 

As generation, particularly the Kinder Morgan power plant, began ramping toward full output, 
the 46 kV system was restored in the affected area. By 10:05 p.m. on August 14 all 46 kV lines 
had been energized and all load was returned to service. Automatic relays were checked and it 
was determined that the relays had not tripped as initially believed. The field personnel 
dispatched from surrounding areas were returned to their respective headquarters. 

The ties between METC and Lansing BWL remained closed from the METC end throughout the 
event. At 7:45 p.m. on Thursday, August 14, Lansing BWL closed their end of the ties in 
agreement with METC. 


17 See Section 3.5.2 for an explanation of black start procedures. 
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According to Consumers Energy's Outage Management System, up to 11 8,400 customers were 
out of service during the 4:00 p.m. through 10:00 p.m. timeframe on August 14 th . 

A restart of Campbell 3 was attempted at 7:44 p.m. but was aborted when the unit experienced 
water hammer two minutes after turning on steam. This damaged a number of piping hangers in 
the plant, which required repair, delaying the units return to service. As information on 
Campbell 3’s status and repair time became known, and with the uncertain status of the 
transmission system. Consumers Energy issued a request that customers continue to conserve 
electrical use into Friday, August 15. 

With the southeastern portion of the Consumers Energy/METC system returned to a normal 
situation, except for the Whiting Generation and the 138 kV ties to ITC and NIPSCo, seventy- 
five percent of the power supply to the affected area was coming from the Kinder Morgan power 
plant and transmitted on the Leoni-Beecher (Jackson to Adrian) and Leoni-Parr Road-Whiting 
(Jackson to Monroe) 138 kV Lines. The remaining twenty-five percent of the power supply was 
from the Verona Substation in Battle Creek and was transmitted on the Verona-Batavia 138 kV 
Line (Battle Creek to Coldwater). These three 138 kV lines, now critical to supply power to the 
recently restored area, were heavily loaded but within applicable continuous capability limits. 

At 10:30 p.m. the Leoni-Beecher 138 kV Line tripped and did not re-close at Beecher due to loss 
of station power. This resulted in large flows on the remaining two critical 138 kV lines, causing 
them to open at their source ends. The system within the subject geographic area was then in 
nearly the same state as it was following the primary 4:09 p.m. outage. 18 Immediately, a similar 
restoration plan to the primary outage was executed. The 138 kV system was restored by 12:55 
a.m. Friday, and the 46 kV system along with all of the connected customers was restored by 
1:35 a.m. Friday. 

During restoration efforts field personnel were dispatched to patrol the Leoni-Beecher 1 38 kV 
Line. Relays protecting this line were remotely interrogated and a suspected fault location was 
identified. This information was passed on to the field crews as a place to start their patrol. Due 
to darkness and foggy/hazy conditions, the source of the fault was not located. When that line 
was returned to service at 1 1 :00 p.m., it was given a derated capability equal to its historically 
highest sustained power flow in order to avoid further trips or failures. 

According to Consumers Energy’s Outage Management System, up to 70,100 customers were 
out of service during the 10:00 p.m. August 14 th through 6:00 a.m. August 15 th timeframe. 

The morning weather forecast for August 1 5 was reviewed and with the capacity available at that 
time, Consumers Energy was expected to be short of ECAR operating reserve requirements. 
Internal company load reductions were ordered, and the public was asked to conserve all day. 
Specific customers were contacted to request voluntary curtailments and an estimated 400 MW 
of reduction was obtained. At 12:45 p.m., Detroit Edison gave Consumers Energy permission to 
use its unused Ludington capacity to meet load. By 4:00 p.m. on August 15, rainstorms 
occurring across the Consumers Energy service area reduced load, which allowed the widespread 
general public conservation to be discontinued. 


18 However the night-time load was considerably less then the afternoon load. 
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Several reliability concerns were handled by Consumers Energy personnel over the next two 
days. These included problems in adhering to the derated capability of the Leoni-Beecher line, 
the clearance status of 138 kV lines located within the affected area, and large power flows 
between the METC and ITC systems. In addition, continued hot weather, unit outages caused by 
the event, and uncertain power availability to supplement Consumers Energy’s own internal 
generation led to a forecast of a deficiency in Consumers Energy’s operating reserve. 

Adhering to the derated capability of the Leoni-Beecher line resulted in Consumers Energy’s 
System Control curtailing load. As loads came up on Friday morning, power flow on the Leoni- 
Beecher line began to exceed the derated capability established the previous night, risking a line 
trip again and potentially another outage. Two actions were taken to adhere to the derated 
capability. All customers in the affected area were asked to curtail power usage and load 
management procedures were put into place. This resulted in a forced outage of industrial 
customers in that area and a reduction of approximately 16 MW of load. Additionally the Raisin, 
Tecumseh Products, La Salle, and Erie 46 kV lines were forced out of service. This resulted in a 
reduction of approximately 40 MW of load. According to Consumers Energy’s Outage 
Management System, up to 17,500 customers were out of service during the 7:00 a.m. through 
noon timeframe on August 15 th. 

With the return of the first Whiting generator at 9:30 a.m. on August 1 5, the 46 kV lines were 
restored to service. However, continued restriction of the related large customers was 
maintained since loading on the Leoni-Beecher line remained at approximately ninety percent of 
the derated capability. This restriction was imposed until generation at Whiting stabilized on 
Saturday and all related industrial customer load restrictions were lifted. 

All 138 kV lines contained within the affected area, especially the 138 kV lines deemed critical, 
were patrolled on Friday, August 15 in an effort to avoid a repeat of the Leoni-Beecher line trip. 
After all items identified as possible concerns were resolved, the remaining ties between 
Consumers Energy/METC system and Detroit Edison/ITC were dosed by 1:05 p.m. on 
Saturday, August 16. 

The final concern was possible separation between the Consumers Energy/METC and Detroit 
Edison/ITC systems due to any one of three single-contingencies involving tie lines between the 
two systems on Saturday. 19 With Detroit Edison being generation deficient, it was dependent 
upon the Consumers Energy/METC system interface for power supply, particularly in the thumb 
area of Michigan. With power flows between the two systems reaching the 3,000 MW range, 
analysis indicated a single contingency would load other ties between the Consumers 
Energy/METC system and the Detroit Edison/ITC system above emergency capabilities. This 
could start a cascading outage resulting in separation between the METC and ITC systems. A 
number of items were implemented to prevent this from occurring. On the daily morning 
conference call with METC these concerns were discussed and patrols were ordered for the 
METC portion of the tie lines identified by the analysis. MECS and ITC were notified of the 


19 The term “single-contingency” is used in the industry to indicate a situation resulting from the outage of a single 
unit or transmission line currently in operation. The concept is to plan for operation of the system in a manner that 
will allow it to continue to function in the event that something unexpected happens. 
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patrol activities. An additional call to discuss these issues was scheduled with Consumers 
Energy, Detroit Edison, MECS, METC and ITC for 2:30 pm. After this call, power flows on 
these ties decreased within applicable limits as generation within the Detroit Edison/ITC system 
increased. 

Concerns were monitored and discussed via conference calls throughout the weekend as 
preparations were made for Monday business. 


Section 3.5.2: Detroit Edison 

The Detroit Edison service territory-wide outage invoked the utility’s black start procedures. 
These procedures were initially developed after the 1965 outage and direct all of the available 
field operations staff to the proper locations to support the restoration effort. Given the 
telecommunication and traffic issues that occurred immediately after the incident, these 
procedures saved valuable time restoring the system. 

Chart 3.7 



Detroit Edison was faced with the difficult prospect of restarting its entire collection of 
generation facilities as listed in Chart 3.6 from a near black start position. This term is 
referenced in textbooks and technical literature to describe a situation in which a power plant is 
being brought online with a transmission system that is not energized. While utilities maintain 
procedures for this condition, it is not frequently encountered. A black start condition is 
particularly challenging because for most generation, power is required to begin the process. 
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The turbine must begin to turn to start fluid flow, and steam generation, water flow and 
compressed air are also required during start up. These functions require electrical power, and 
that power must be made available either through the transmission system or local stand-by 
generators. 


Chart 3.8 


Unit 

Initial inspection Results 

Synchronizing 

Time 

Full load 

Comments 

Trenton Channel 7 A 
(110MW) 

No issues found during 
initial inspection 

August 15, 21X13 
0916 hrs 



Trenton Channel 8 
(100MW) 

No issues found during 
initial inspection 

August 15. 2003 
0330 hrs 

First attempt on 
August 14. 2003 at 
2340 hrs 

August 15, 2003 
1430 hrs 

r* *, .• 

M;0M 

Trenton Channel 9 
(520MW) 

No issues found during 
initial inspection 

August 19, 2003 
1405 hrs 

August 20, 2003 
0550 hrs 

69 SS Transformer available for 
unit restart at 0400 hrs on 

August 15. 

Turbine vibration was reduced 
with a balance shot Load 
continues to be limited to 475 
MWs due to high vibration. 

Monroe 1 (77GMW) 

Unit was in an outage at 
the time of the event 

August 18. 2003 
0359 hrs 

August 18, 2003 
1156 hrs 


Monroe 2 (750MW) 

Four failed rupture discs 
were found/repiaced, loss 
of control air on generator 
seal oil system valves 
resulted in oil in the 
generator 

August 19, '2003 

01 14 hrs 


• "m T.-vif J 
, - 

Monroe 3 (750MW) 





Monroe 4 (775MW) 




ililjsgfiiESi 

River Rouge 1 
(225MW) 

Unit was in an outage at 
the time of the event 

■emIS ■ 

2 

■ ■ 

1 1 

River Rouge 2 
(247 MW) 

Unit was in an outage at 
the time of the event 

August 16, 2003 
1422 hrs 

E 

HP collector repairs were being 

conducted at the time of the 
event 

River Rouge 3 
(272MW) 

The MTG was placed on 
turning gear to straighten 
the shaft. 


III 

The bearings were found to be 
damaged during an attempted 
start. This unit will require 
several weeks to repair and is 
not expected to be operational 
again until fate September. 


Another factor affecting return to service of generation units is synchronization. When 
connecting a power plant to the electrical system, the frequency, voltage and phase ol the plant’s 
output must be made to match the rest of the electrical system. If this is not the case, the shaft 
and equipment on the shaft could be severely damaged and require months to repair. 
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Large generators and their associated equipment require a ramp-up period before they can be 
placed under full load conditions. The thermodynamic aspects of operating power plants require 
operators to maintain the plant at a low output level for a period of time to allow the components 
to reach thermal stability. Once this point is reached, the plant can be pushed to full load. 

Chart 3.8 (continued) 


Unit 

Initial Inspection Results 

Synchronizing 

Time 

Full Load 

Comments 

Bei!e River 1 
(625 MW)* 

Three failed rupture discs 
were found/replaced 

Attempted 
synchronization on 
August 15 at 21 M 
hrs - Lack of field 
excitation prevented 
synchronization 


Inspection revealed a failed 
exciter. Exciter problems were 
corrected and the unit was 
returned to service on August 
14,2003 at 1317 hrs 

Belie River 2 
(S35MWT 

Three failed rupture discs 
were fcund/replaced 

August 16, 2003 
1245 hrs 

August 17, 2003 
0045 hrs 


Si. Clair 1 (1S3MW) 

No issues found during 
inspection following the unit 
trip 

August 15, 3303 
1924 hrs 

August 17,2003 
1000 hrs 

Unit remained on the 120 kv 
system during the initial event - 
Tripped offline during switching 
operations on August 14, at 

1032 hrs 

Following restart load was 
limited due to a ground in the No 
Load/Low Load trip switch 

St. Clair 2 (1G2MW) 

No issues found during 
initial inspection 

August 15, 2003 
1223 hrs 

August 15, 2003 
2000 hrs 


St. Clair 3 (171 MW) 

No issues found during 
inspection followinq the trio 

August 15, 2003 
0541 hrs 

August 15, 2003 
1300 hrs 

Unit remained on the 120 kv 
system during the initial event - 
Tripped off line during switching 
operations on August 14, at 

1834 hrs 

St. Clair 4 (158MW) 

No issues found during 
initial inspection 

August 15, 2003 
1645 hrs 

August 15,2003 
2300 hrs 


St. Clair6 (321 MW) 

Initial inspection revealed a 
boiler water circulating 
pump overheated 

August 16, 2003 
0948 hrs 

August 16, 2003 
1500 hrs 

Boiler watef circulating pump 
overheated and had to be 
replaced prior to start-up 

St. Clair 7 (450MW) 

Failed rupture discs 
found/replaced 

August 15, 2003 
1901 hrs 

August 16, 2003 
1300 hrs 

Difficulty with turbine turning 
gear, precipitators and ash 
handling 


Failure to follow this procedure can substantially damage key equipment on the generating unit 
and require months or years to correct. 

The Harbor Beach power plant (103 MW) tripped off-line during the event on August 14 but was 
restored to service within a few hours. Greenwood Energy Center (785 MW) was restored to 
service mid-day on August 15. Unit 15 (150 MW) and unit 16 (65 MW) at Conners Creek 
power plant were restored to service by mid-day on August 16. Ludington Pumped Storage 
Facility remained available throughout the restoration effort. Fermi 2 (1130 MW) tripped off- 
line during the event and was restored to service on August 20, 2003. The time period to start 
Fermi was longer due to the procedures required for nuclear power plants. A timeline for the 
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remaining major plants is provided in Chart 3.8 20 . In addition to the work described in Chart 3.8, 
the Dean Power Facility (an independent power producer) was returned to service at 8 : 1 5 p.m. 
August 14 and Detroit Edison personnel resolved issues at various peaking facilities to provide 
950 MW of additional capacity on August 15. 

In addition to Detroit Edison’s generation, purchases from outside the service territory in 
coordination with independent power producers were crucial to a timely restoration. This was an 
effective method of restoration, but expensive. Some purchases were $87/MWb or more. A 
normal summer on-peak purchase cost would be around $50/MWh. Detroit Edison also 
coordinated filling the Ludington Pumped Storage Facility with Consumers Energy during the 
night of August 14 to prepare for load restoration on August 15. To prevent rolling blackouts in 
Consumers Energy’s territory, Detroit Edison allowed Consumers Energy to use some of this 
stored energy on August 15. 

Detroit Edison’s System Operations Center (SOC) was monitoring the steady state condition of 
the transmission system at the time of the blackout. There were no indications or notifications of 
impending problems prior to the blackout, which occurred in a matter of seconds. After the 
blackout, the first actions taken were to determine the state of the system. Within a few minutes, 
the system supervisors determined the following: 

• System frequency was zero. 

• St. Clair Power Plant had two units running in stable condition and serving an island of 
approximately 89 MW. 

• A group of customers were being served in the thumb through an interconnection. 

• A group of customers in the Pinckney area were being served by the Majestic 
interconnection. 

• Harbor Beach Power Plant had tripped, but was operational. 

• Several interconnections were still energized and available to support the restoration 
effort. 

In addition to the field operations staff, six additional system supervisors, three engineers and 
several application engineers joined the available SOC staff within an hour to support the 
restoration effort. Shortly after the incident, numerous members of senior management arrived 
in SOC to assess the incident and ensure that the proper resources were available. Within 3 
hours, additional support was made available from DTE Energy’s generation operations and 
major account services organizations. 

The 1965 plan, which had since been updated on numerous occasions, was used as a guideline by 
SOC. The SOC team broke up into north and south teams. The intent was to begin restoring the 
120kV transmission loops around the service territory and provide a source of electricity for 
power plants to use to start and synchronize. In the early hours of the blackout, the focus was to 
restore a source of power to the plants and switch load back onto the electrical system in a safe 


20 Chart 3.8, the Generation Restoration Timeline, was prepared by Detroit Edison shortly after the blackout and 
provides a contemporaneous accounting of the status of each of the company’s generating units immediately prior 
to, during and subsequent to the blackout. We note that the data for the return to service of Belle River 1 is in error. 
The unit actually returned to service on August 24 at 1:17 p.m. Otherwise we believe that the information is correct. 
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and orderly manner. The effort was complicated by numerous external factors. Most 
significantly, releases of hydrocarbons and evacuation around the Marathon Oil refinery blocked 
several plans the SOC team formulated to reach some of the southern power plants. The team 
eventually achieved this goal with police support. 

The service territory-wide outages include the following counties: Wayne County, Oakland 
County, Monroe County, Macomb County, 1 0% of Lapeer County, 25% of St. Clair County and 
the eastern half of Livingston County - all which had lost power within a 5 minute period. The 
following counties were not affected because the Atlanta-Kam-Thetford 120 kV and the 
Hemphill-Hunters Creek 120 kV interconnections to Consumer Energy remained in service: 
Sanilac county, the northern 90% of Lapeer County, and the northern 75% of St. Clair County. 

In the southern portion of the system at Majestic station, the 345 kV interconnections with 
Consumers Energy and the 345 kV interconnections with FirstEnergy remained in service, which 
allowed the western half of Livingston County to remain in service following the event. 

After the blackout, Detroit Edison found it was in the following condition: (1) the St. Clair Units 
1 and 3 were available but isolated in an area near the facility; (2) the Atlanta -Kam-Thetford 
interconnection to Consumers Energy/METC was energized; (3) all 345 KV interconnections to 
Majestic Station were closed and energized; (4) the Allen Junction-Majestic-Monroe 
transmission line was energized into the Monroe Power Plant; and (5) the Majestic lines were 
feeding and carrying Madrid and Genoa Station load. At this point all plants were ordered to 
initiate black start procedures. 

In its restoration process Detroit Edison focused on first energizing a 120 kV path to connect 
power plants and re-establishing the available inter-connections to neighboring utilities. The first 
step taken was to restore power to the Harbor Beach plant and Greenwood Energy Center. In the 
northern part of the system, the Atlanta-Kam-Thetford 120 kV and the Hemphill-Hunters Creek 
120 kV interconnections were utilized to provide restoration paths to Harbor Beach and St. Clair 
power plants. This allowed Harbor Beach generation to start and load was restored in the 
remaining portions of Lapeer County by 8:00 p.m. Also, a source of power to start the St. Clair 
plant and Dean generation was established. The St. Clair 120 kV bus' 1 was restored at 8:15 p.m. 
and the St Clair peakers were started. 

During the same period on the south side of its service territory, Detroit Edison was able to 
isolate a path between the Monroe Power Plant, Brownstown Station, Fermi nuclear plant and 
the Trenton Channel plant. Between 10:00 p.m. and 10:30 p.m., the Allen Junction-Majestic- 
Monroe interconnection between Detroit Edison and FirstEnergy was used to establish a 
restoration path from Monroe to Brownstown to Fermi. An additional path was isolated from 
Brownstown to Navarre Station to Waterman Station. This provided Detroit Edison with the 
ability to provide station power to these units in preparation of the process of restoring them to 
service. Just before midnight the lines to Trenton Channel from Brownstown were energized. 

Early in the morning of Friday, August 15, Detroit Edison restored the Pontiac-Hampton 
interconnection, which further strengthened Detroit Edison’s interconnection with the 
Consumers Energy/METC system and allowed a restoration path to be established to the 


21 “Bus" is short for bus bar, a conducting bar that carries heavy currents to supply several electric circuits. 
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Greenwood power plant, the Belle River power plant, and St. Clair Units 6 and 7. The 
Essexville-based generation of Consumers Energy could now assist in the restart. The restoration 
of interconnections with the Consumers Energy /METC also allowed Detroit Edison to use the 
Ludington Pumped Storage Facility and purchases from AEP to assist in the restoration process. 
The Belle River Power Plant-Greenwood-Pontiac Station connection was also restored. This 
connected the Detroit Edison units at Belle River and the Greenwood Energy center to the 
system. At this point the Greenwood peakers were started. 

Around 3:00 a.m., the 120 kV Trenton Channel-Airport line was re-energized to restore power 
to Metro Airport, which began restoration of critical facilities. 

At 3: 18 a.m., Remer Station was restored from the St. Clair generation facility to allow Dean 
Generation to restart. This connected an additional 300 MW of generation to the Detroit Edison 
system. 

At 3:30 a.m., Detroit Edison energized Brownstown-Navarre-Waterman line. The River Rouge 
Power Plant could not be reached due to Marathon Oil Refinery incident, which required 
evacuation of the area surrounding the refinery. 

Between 3:55 a.m. and 4:43 a.m., the Belle River-St. Clair 345 kV line was restored, along with 
the Belle River- Jewel interconnection. This action tied the Belle River and St. Clair power 
plants into the system, so they could begin supporting the restoration activity. By 5:00 a,m., the 
restoration of the Pontiac-Hampton 345 kV interconnection, and the interconnection to the 
Greenwood plant, the Belle River power plant, and St. Clair Units 6 and 7 allowed the 
restoration of approximately 10% of the load in northern Macomb and Oakland Counties and 
approximately 20% load in St. Clair County. 

Restoration of interconnections from St. Clair plant through Stephens station to Northeast station 
allowed Northeast peakers to feed into the system. This allowed the load to be picked up in the 
remaining portions of St. Clair County and approximately 70% of Macomb County. Also by 
5:00 a.m., additional paths were established which allowed power to be restored to the northern 
1/3 portion of Monroe County and Southern 1/3 portion of Wayne County. 

Between 7:00 a.m. and 8:00 a.m., a path from Navarre to Waterman was established to provide a 
source of power to get the Delray peakers started. This path was then extended to provide the 
River Rouge power plant with a source of power. As these paths were energized a small amount 
of load in the City of Detroit was restored. Additional lines and substations were energized by 
noon to pick up additional load in the City of Detroit. Paths were energized between Majestic 
station and Pontiac station to connect the southwest and northwest portions of the system. Paths 
were energized between Waterman and Northeast stations to connect the northern and southern 
portions through the City of Detroit. 

The Blackfoot Station-Madrid Station line was re-energized at 8:55 a.m. and at 9:30 a.m. Detroit 
Edison closed the ring bus at Pontiac. These actions coupled with the restoration of the Bismark- 
St. Clair line set the stage for restoration of customers in the Pontiac area. 


59 



436 


The restoration of distribution load continued with Grayling Station and Malta Station at 9:35 
a.m. Further stations were restored between 9:00 a.m. and noon, when lines were energized into 
Cato Station, St. Antonie Station and Zug Station picking up more load in the City of Detroit as 
Detroit Edison continued its practice of closing in whole distribution centers without isolating 
individual circuits 

At 10:5 1 a,m., Stephens-Victor was closed and tied to the St. Clair facility - this action allowed 
the restoration of distribution load at Victor. At 1 1 :55 a.m., Detroit Edison personnel energized 
the Lincoln 120 kV buses and transformers. After the evacuation requirements associated with 
the Marathon Oil Refinery incident were lifted, operators were able to return to the Navarre 
station and restore all of Navarre between noon and 1:30 p.m. 

Just after noon the Caniff 120 kV buses from Northeast and the Sterling 120 kV buses from 
Jewell were energized. At 12:42 p.m., Detroit Edison energized the Sterling 40 kV buses, the 
Pont Wixom-Wixom connection and then Malta-Red Run at Malta. The preparation to restore 
power to Detroit continued with the energizing of Jewell-Stephens at Jewell at 12:50 p.m. and 
the energizing of Northeast-Stephens at Northeast at 1:02 p.m. 

At 1:24 p.m., Detroit Edison closed the 345 kV ring at the St Clair power facility fully 
connecting the facility to the 345 kV system. 

Between 1 :30 p.m. and 2:30 p.m., all of the 120 kV system at the River Rouge power facility 
was restored. The River Rouge facility was now positioned to fully assist in the restoration of 
the Detroit area. 

The Jewell-Thetford interconnection was tied in at 1:38 p.m., which further strengthened the 
interconnection with Consumers Energy/METC. 

At 1 :45 p.m., the preparation to restore substantial portions of the City of Detroit load continued 
as Stephens was tied to the 120 kV system (this tie would be energized at 2:30 p.m.), Caniff- 
Stephens was energized at Stephens, and the Mack 24 kV buses were energized. The Erin 120 
kV system was energized from Stephens via Erin at 1 :53 p.m. 

For the most part, these actions were preparatory. The restoration of distribution load began in 
earnest at 2:30 p.m. when the distribution load at Mack was restored. Further activity at the sub- 
transmission level continued in order to prepare for additional load restoration in the Detroit area 
with the energizing of the Northeast 24 kV buses and the Lincoln 24 kV buses and transformers. 
Activity continued at the transmission level as Bismarck-Stephens and St. Clair-Stephens were 
energized and Caniff-Stephens at Caniff and Mack-Northeast at Mack were closed. 

Bloomfield Point at Pontiac (230 kV) was energized at 2:45 p.m. and the Bloomfield 120 kV 
buses were energized via Bloomfield-Troy at 3:00 p.m. The 345 kV ring at Bismarck was 
closed during the same time period. The Jewell buses and Spokane 120 kV buses via Jewell-St. 
Clair were energized at 3:20 p.m. and Bloomfield-Pontiac at Bloomfield was closed at 3:44 p.m. 
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At 4:00 p.m. the distribution load was restored at the Jewell sub-station. Shortly after that, 
Detroit Edison energized the Saturn and Frisbie interconnection, the Sloan bus, and Alpha bus 
from Sterling. 

The Mack-Voyager line at Mack was energized at 4:30 p.m., followed by the Chestnut 120 kV 
buses from Lincoln. The Chestnut-Red Run line was closed at Red Run at 5:10 p.m. and the 
distribution load at Red Run was restored. 

Detroit Edison energized the Essex 24 kV system at 5:30 p.m., which restored distribution load 
in the City of Detroit. At 5:40 p.m., Detroit Edison energized the Apache 120 KV and the 
Seneca 120 KV buses, followed by the closure of Essex-Voyager. At approximately 5:30 p.m., 
the Detroit Water Works was energized. 

Paths were energized from Northeast station towards Bloomfield station at the same time paths 
were energized from Pontiac station to the Bloomfield station. This allowed load to 50% of 
Oakland County to be restored at 7:00 p.m. During this period of time, paths were also 
energized from Northeast station to the City of Detroit. 

Restoration efforts required coordination between restart of generating units and transmission 
operations. 1TC was in contact with Detroit Energy’s SOC and Emergency Headquarters during 
the restoration effort, but was dependent on Detroit Edison personnel to complete restoration 
work because the Detroit Edison transmission system was recently sold to ITC and the sales 
agreement included a maintenance agreement until January 2004. Consequently, Detroit 
Edison’s SOC restored the transmission system, and the utility’s System Planning and 
Engineering organization arranged for visual and thermal inspections of the transmission system. 

Because of concerns for the continued reliable operation of the transmission system, the SOC 
team requested transmission inspections on August 16 at approximately 2:00 a.m. This initial 
request was delayed until storm conditions abated. On August 16, ITC requested inspections as 
well. Detroit Edison used visual and thermal inspections performed from the ground and the air. 
ITC and Detroit Edison jointly prioritized the inspections. 

On August 17, Detroit Edison began visual inspections of all interconnections using helicopter 
patrols and focused on detecting mechanical defects. The patrols were completed with no major 
mechanical problems identified. Also on August 17, Detroit Edison personnel began 
thermovision ground patrols. The focus of this effort was to perform close inspections of critical 
equipment in the 345 kV and 230 kV switching yards, not easily seen from the air. All 120 kV 
equipment contained in these yards was also inspected. One helicopter team and two ground 
teams were assigned on August 17. 

On August 18, two thermovision-equipped helicopters began patrols, which allows them to look 
for overheated connections on the interconnection ties. Fifteen inspections were completed by 
Tuesday, August 1 9. On Tuesday, August 1 9, ITC supplied two additional thermovision- 
equipped helicopters. All four helicopters patrolled the 345 kV corridors between stations 
looking for overheated connections. Patrols of most of the 345 kV and 230 kV critical 
equipment and the thumb area north of 1-69 were completed on August 19. On Wednesday, 
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August 20, the remaining 345 kV and 230 kV patrols were completed before moving on to the 
120 kV stations, as prioritized by ITC based on the size and the complexity of the station. All 
helicopter thermovision patrols were completed by 4:00 p.m. Thursday, August 21, 2003. 
Approximately 1,500 miles of transmission line were inspected in five days. 

On Wednesday August 20, the ground patrols for the remaining 120kV stations began and were 
completed by August 30 as requested by ITC. Over 50 stations were inspected from the ground 
in 14 days. 

After the event on August 14, work crews were held to support the restoration effort, but the 
major portion of the distribution effort began on August 15 as the generation and transmission 
systems became functional again. Distribution support was critical to ensure that load was 
properly restored to maintain system stability. The effort was frustrated by the “In Service 
Application” not being available due to the blackout. 22 The effort was also constrained by the 
difficulties of maintaining fuel for corporate and employee vehicles and the sporadic availability 
of the communication systems. The unavailability of the “In Service Application” required work 
orders to be faxed to regional operations centers. In some cases, the communication issues 
required work crews to drive back to the service centers to acquire their next set of work orders. 
The August 16 storm also complicated the restoration effort. Approximately 500 distribution 
engineers, lineman and other employees were involved in the distribution restoration efforts. 

Early in the outage Detroit Edison established a communication link with the Detroit Water and 
Sewerage Department (Detroit WSD). Without water, sanitation issues became a major problem. 
In addition, hospital operating rooms could not function nor could they provide adequate care in 
emergency situations. Detroit Edison worked closely with the Detroit WSD Director to identify 
the priority pumping stations and worked to restore the four units that would have the most 
impact on operations. These were restored throughout Friday with the final critical station 
restored by early Friday evening. 


Section 3.5.3: Lansing Board of Water and Light 

At 6:02 p.m., the Lansing BWL began to establish a cranking path to allow the use of Consumers 
Energy generation to assist in the restoration of station power to its generation facilities. By 6:30 
p.m., a cranking path had been established to Eckert Station and the Erickson facility. By 10:07 
p.m., the Erickson generation was back on-line and at 10:19 p.m. Lansing BWL restored its first 
group of customers. Most of the Eckert Station units returned to service just after midnight, with 
Unit 5 returning at 3: 16 a.m. At 4:21 a.m., on August 15, Lansing BWL restored service to the 
last circuit of customers. The time period from the start of the event to full restoration was just 
over 12 hours and 10 minutes. 


22 In Service Application is a computerized field force coordination (unction used by Detroit Edison to prioritize and dispatch its 
repair crews. 
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Section 3.6: Review and Analysis 

In our opinion, the restoration after the blackout by The Detroit Edison Company, Consumers 
Energy Company, and the Lansing Board of Water and Light was fully acceptable. 

The restoration effort was made possible by the automatic opening of the interconnections 
between Consumers Energy/METC and Detroit Edison/ITC, which protected the transmission 
equipment of both systems from damage. During the initial phase of the restoration process, this 
was the only electrical connection that Detroit Edison had with other power supplies. This 
transmission link also connected Detroit Edison with its generation facilities at Ludington, 
another essential tool in the restoration process. In addition, Consumers Energy provided 
cranking power to assist the Lansing BWL in restoring service. 

Michigan systems had options that could have been exercised and that may have prevented the 
spread of the blackout into Michigan. Detroit Edison, given adequate warning, could have 
unloaded the METC and ITC interconnections between itself and Consumers Energy in advance 
of the event. An unloaded interconnection may have weathered the power surge and spared the 
Detroit Edison system from the blackout. 

In the heavily interconnected power system of today, almost every major action taken by an 
entity connected to the grid is capable of impacting the electric system within Michigan. The 
events that precipitated the blackout occurred beyond the view of the system operators in 
Michigan. With no advance warning, the operators were unable to take any action to protect 
customers from the impending outage. As discussed more completely in Part II, adequate 
reliability standards and an effective enforcement organization are required to ensure the 
reliability of Michigan systems. 

Additionally, it appears likely that two factors hampered the restoration efforts. First, the 
computerized “In Service Application” system used to dispatch and coordinate personnel was 
inoperable. This system was clearly not designed for a blackout of this magnitude, which 
required the first use in Michigan of widespread black start procedures. The lack of emergency 
power for this system required additional time and effort for restoration. The “In Service 
Application” process has performed satisfactorily during other, smaller outages. In our opinion, 
Detroit Edison should conduct an analysis of the “In Service Application” process to determine 
what modifications are warranted in light of the experience gained in this restoration effort. 
Detroit Edison should report to the Public Service Commission on the results of its analysis. 

The second factor was the failure of rupture disks at four of the Detroit Edison generating units. 
The failed rupture disks slowed the pace of restoration. Six of the 13 plants without failed 
rupture disks were returned to full load on August 15 and three more on August 16. Conversely, 
none of the plants with failed rupture disks were returned on August 15 and only one on August 
16. 

However, the presence of failed rupture disks does not, in itself, indicate a problem that needs to 
be corrected. Rupture disks are a design feature - they are intended to fail under certain 
conditions, thereby avoiding more serious damage to equipment. Thus, there is a trade-off 
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involving the avoidance of major damage and the potentially more frequent occurrence of 
outages due to failed rupture disks. An analysis of the operation of rupture disks and the trade- 
offs involved requires specialized engineering expertise, which the PSC Staff does not possess. 
Accordingly, we recommend that Detroit Edison analyze the operation of the rupture disks on its 
units, including a comparison with the operation in other utility systems affected by the blackout, 
to determine whether any changes are warranted. Detroit Edison should report the result of its 
analysis to the Commission. 


Section 3.7: Recommendations 

We make the following recommendations: 

1 . That Detroit Edison conduct an analysis of the “In Service Application” system to 
consider modifications or alternatives that would function more effectively in the 
event of a similar blackout and report the results of its analysis to the Commission. 

2. That Detroit Edison conduct an engineering analysis of the operation of the rupture 
disks to determine if any modifications are warranted and report the results of its 
analysis to the Commission. 
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PART IV 


EMERGENCY PLANNING AND RESPONSE 


Section 4.1: Introduction 

This part of the report addresses specific response measures initiated in Michigan as a result of 
the blackout of August 14, 2003. It further makes recommendations on specific actions that 
should be considered as a means to improve the response to a power outage of this type in the 
future. It should be remembered that there has never before been a blackout of this nature in 
Michigan. As we look for solutions to prevent a reoccurrence, the focus should be on reducing 
risks and vulnerability to all hazards. While some problems did develop, to a considerable 
degree the response went as planned and worked well, and more serious problems were largely 
averted. 

The sections of this report present a description of the roles and responsibilities of the PSC and 
various other State and federal agencies (Section 4.2); a discussion of the response by PSC, its 
Staff, and the State of Michigan Emergency Management Team (Section 4.3); and lessons 
learned and recommendations for improving future State responses (Section 4.4). 

Michigan State Emergency Operation Center 
August 14, 2003 
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Press Conference August 15, 2003 



Section 4.2: Roles and Responsibilities 

To understand the sequence of events and actions taken, it is important to understand the roles 
and responsibilities of the PSC and its Staff for energy emergency preparedness and response. 
The PSC is charged with assuring that sufficient energy resources are available to Michigan’s 
citizens and businesses at competitive prices. 2 ' 1 As part of this charge, the Commission has two 
separate but related responsibilities. The first is energy emergency preparedness. If a supply 
problem develops with natural gas or electricity, the Commission has adopted through rules and 
orders the procedures that a utility will use to respond, which include provisions for reporting on 
its actions to the Commission. If au energy emergency requires mandatory State action, the 
Governor, upon recommendation of an interdepartmental Energy Advisory Committee or at her 
own initiative, may declare a State of Energy Emergency under 1982 PA 191, as amended (MCL 
10.81). The Governor may order mandatory actions following such a declaration. 

Second, if the situation worsens, or another event such as a tornado, flood, or terrorist attack 
occurs, the Governor can declare a Stale of Disaster. In this case, the primary responsibility of 
response efforts shifts to the Emergency Management Division (HMD) of the Michigan State 
Police (MSP), through which the PSC Staff would provide a support function. In addition, each 
department of state government has designated an Emergency Management Coordinator (EMC) 


y MCL 460.901 
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who represents the department and coordinates departmental resources that may be needed to 
respond to any given event, PSC Staff have been assigned to serve this function in the 
Department of Consumer & Industry Services (CIS, but soon to become the Department of 
Licensing and Economic Development), in addition to the Commission’s responsibility for 
energy-related matters. 


Section 4.2.1: Energy Emergency Act 

The Energy Emergency Act grants the Governor broad powers in the event of a Declaration of an 
Energy Emergency. These include the following, as described in the Act: 

“10.84 Powers of governor during energy emergency. 

Sec. 4. During an energy emergency, the governor may do all of the following: 

1 . Order specific restrictions on the use and sale of energy resources. Restrictions 
imposed by the governor under this subdivision may include: 

a. Restrictions on the interior temperature of public, commercial, industrial, and 
school buildings. 

b. Restrictions on the hours and days during which public, commercial, industrial, 
and school buildings may be open. 

c. Restrictions on the conditions under which energy resources may be sold to 
consumers. 

d. Restrictions on lighting levels in public, commercial, industrial, and school 
buildings. 

e. Restrictions on the use of display and decorative lighting. 

f. Restrictions on the use of privately owned vehicles or a reduction in speed limits. 

g. Restrictions on the use of public transportation, including directions to close a 
public transportation facility. 

h. Restrictions on the use of pupil transportation programs operated by public 
schools. 

2. Direct an energy resource supplier to provide an energy resource to a health facility; 
school; public utility; public transit authority; fire or police station or vehicle; 
newspaper or television or radio station for the purpose of relaying emergency 
instructions or other emergency message; food producer, processor, retailer, or 
wholesaler; and to any other person or facility which provides essential services for 
the health, safety, and welfare of the residents of this state. 

3. By executive order, suspend a statute or an order or rule of a state agency or a specific 
provision of a statute, rule, or order, if strict compliance with the statute, rule, or order 
or a specific provision of the statute, rule, or order will prevent, hinder, or delay 
necessary action in coping with the energy emergency. The governor may not 
suspend a criminal process or procedure or a statute or rule governing the operation of 
the legislature. At the time of the suspension of a statute, rule, or order or a specific 
provision of a statute, rule, or order, the governor shall state the extent of the energy 
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shortage and shall specify the provisions of a statute, rule, or order which are 
suspended, the length of time for which the provisions are suspended, and the degree 
to which the provisions are suspended. A suspended statute, rule, or order shall be 
directly related to an energy emergency.” 


Section 4.2.2: Energy Advisory Committee 

Section 10.82 of the Energy Emergency Act provides for an Energy Advisory Committee (EAC), 
which is responsible for notifying the Governor of an impending energy emergency. When the 
EAC determines that an energy emergency is imminent based on information from the PSC and 
other sources, the Governor is informed and may respond by declaring a State of Energy 
Emergency. The monitoring activity to forewarn of the potential for an energy emergency is 
provided by the Michigan Energy Appraisal 24 , which is issued twice a year in the spring and fall 
by the PSC. In recent years, the Commission has also directed Consumers Energy, Detroit 
Edison and American Electric Power to report on how they plan to meet peak summer needs, and 
PSC Staff has held weekly conference calls with the utilities over the summer months to monitor 
supply and demand conditions. The Governor may also declare an energy emergency on her 
own initiative. The EAC is comprised of the Directors of the Departments of CIS, Agriculture, 
Community Health, Transportation, and MSP. It is chaired by the Chair of the PSC, as provided 
for in Executive Order 1986-17 (MCL 460.901). 

Previous to the August power outage, the EAC had met twice - in the spring of 1979 to respond 
to shortages arising out of the oil distribution caused by the Iranian Revolution, and in June 2000 
in response to the potential gasoline shortage from the Wolverine Pipeline break in Jackson. 


Section 4.2.3: Energy Emergency Management Team 

In the case of an energy emergency or in anticipation of such an emergency, the Chair of the 
MPSC may convene an Energy Emergency Management Team (EEMT). The EEMT will 
monitor developments, prepare assessments, and develop responses. The EEMT consists of 
senior PSC Staff, including representatives from the Commission Operations and Energy 
Operations Divisions. The PSC Chair is responsible for convening the EEMT, assigning tasks to 
its members, and providing information developed by the EEMT to the Governor and EAC. In 
general, the EEMT responsibilities include monitoring developments, preparing assessments, and 
implementing responses on a day-to-day basis. Each member of the EEMT will appoint 
appropriate staff to support EAC work on a priority project assignment basis. 


Section 4.2.4: Michigan Emergency Management Plan 

Michigan Emergency Management Plan (MEMP) is the State’s overall disaster response and 
recovery plan. There is a basic plan as well as specific plans that address specific types of 

M http://www.cis.state.nii.us/mnsc/reoorts/energv/ciiiTeni/ 
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disasters, including: nuclear accidents, enemy attacks, natural disasters, and technological 
disasters. The August 14 power outage fell under the technological disasters category. In each 
of these areas, each department’s roles, responsibilities and authorities are identified. In addition 
to this plan, each department and agency has one or more planning documents which supports in 
further detail the departmental or agency response. This plan is currently undergoing updates 
and revisions. 

It should be noted that, in addition to the PSC responsibilities for coordinating energy 
emergencies, the manager of the Energy Data & Security Section is also the Emergency 
Management Coordinator (EMC) for CIS. The EMC plays a critical role in ensuring that the 
department is capable of implementing the tasks assigned to it in the MEMP before, during, and 
after a disaster or emergency. To that end, each EMC must be concerned with the following 
responsibilities within their department: 

• Developing procedures for carrying out responsibilities assigned by the MSP; 

• Conducting departmental reviews of procedures developed and making revisions 
where appropriate (subject to the approval of department director); 

• Developing necessary support documents (standard operation procedures, resource 
lists, telephone notification lists, etc.); 

• Ensuring staff are aware of and trained for assigned responsibilities; 

• Revising procedures as conditions change (i.e., in organizational structure, 
departmental mission, resource base) in conjunction with EMD/MSP; 

• Conducting a training needs assessment on an annual basis to identify personnel who, 
by virtue of their position or area of responsibility, need to receive emergency 
management training; 

• Getting important emergency management information into the hands of all 
appropriate department personnel; and 

• Developing contacts with federal agencies and private sector organizations within the 
department's sphere of responsibility that could be called upon to assist in disaster 
response and recovery operations; and, representing the department in the SEOC to 
coordinate department response and recovery activities, and to establish 
communications with department field personnel. 

There are two PSC divisions with primary responsibilities in energy emergency planning and 
response activities - the Commission Operations Division (COD) and the Energy Operations 
Division (EOD). The energy emergency responsibilities of these divisions fall into four broad 
categories: 


• Monitor Michigan's energy supply system for the purpose of detecting unusual 
imbalances that may indicate the potential for an energy emergency and advising the 
appropriate state officials in such events; 

• Develop, administer, and/or coordinate energy emergency contingency plans; 

• Act as the communication focal point for federal, state, and local activities related to 
energy emergency planning and management; and 

• Maintain ongoing contact with the petroleum, natural gas, and electric industries 
concerning Michigan's energy situation. 
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During the power outage, a number of PSC Staff provided support at the SEOC. In particular, 
the manager of the Energy Data & Security Section in COD and the safety manager in EOD 
spent considerable time in the SEOC working with the MSP and other state officials monitoring 
the situation and providing consultation for the Governor and the EAC on what Michigan’s rules 
and regulations regarding energy emergencies entailed. 


Section 4.3: Response by the PSC and the State’s Emergency Management Team 

This section addresses PSC, Staff, and State government involvement in responding to the 
outage and its secondary effects. At the onset of the power outage, PSC Staff went immediately 
to the SEOC to manage power outage responsibilities from that site. The SEOC has a backup 
generator and lighting, computer access, and phones were available throughout the emergency. 
Other PSC Staff remained at the PSC offices at Mercantile Way in Lansing for a number of 
hours until it became impractical to continue due to sunset. These Staff members used cell 
phones for communication. 

The PSC provided staffing at the SEOC from 4:30 p.m. on Thursday, August 14, 2003, until 2:30 
a.m. on Friday, August 1 5, returned by 6:45 a.m. that Friday and remained until midnight. 
Staffing on Saturday, August 16, was provided from 7:00 a.m. to 5:30 p.m. During this time, 
there were always at least two PSC Staff members at the SEOC and sometimes three. In 
addition, PSC Chair Lark was on-site for a briefing on each of these days. Although power had 
been restored to all customers by Saturday morning, the concern remained that continued 
voluntary conservation measures were required to reduce demand, while a number of Detroit 
Edison power plants returned to operation. If electric demand exceeded Detroit Edison’s 
available generation, it could have been necessary to implement rotating power blackouts. 
Therefore, PSC Staffs focus on Saturday, August 16, was to monitor Detroit Edison’s ability to 
balance load. 

During and following the power outage a number of issues arose that required response. These 
issues clearly demonstrated the critical interdependencies that exist that support our citizens and 
businesses. The National Strategy for the Physical Protection of Critical Infrastructures and Key 
Assets 25 , issued in February 2003, describes the interdependencies as follows: “The facilities, 
systems and functions that comprise our critical infrastructures are highly sophisticated and 
complex. They consist of human capital and physical and cyber systems that work together in 
processes that are highly interdependent. They each encompass a series of key nodes that are, in 
turn, essential to the operation of critical infrastructures in which they function. To complicate 
matters further, our most critical infrastructures typically interconnect and, therefore, depend on 
the continued availability and operation of other dynamic systems.” 


25 National Strategy for the Physical Protection of Critical Infrastructures and Key Assets, February 2003. See: 
htto://www. whitehouse.gov/ncinb/ohvsical strategy. pdf 


70 



447 


Interdependencies 26 



Following are some of the key issues and actions taken at the SEOC to respond to the outage in 
which many of the critical interdependencies were clearly demonstrated. 


Section 4.3.1: Power Outage Status Assessment, Analysis and Reporting 

This was one of the primary functions of the PSC Staff, with the assistance of Detroit Edison 
personnel who arrived at the SEOC on Friday morning. The SEOC held periodic briefings with 
each of the departmental agencies, providing a summary of events, issues, and problems in their 
area of responsibility. The computer systems normally used by Detroit Edison were not 
available to the company due to the power outage and, therefore, the number of customers that 
were without power could not be provided. Typically, this information is available in situations 
such as storm outages and provides the basis for assessing the severity of the problem and rate of 
restoration. Absent this information, the recovery of the system and restoration of power was 
monitored in three ways. First, Detroit Edison personnel in the control center provided 
information on the load being served. This was a measure (albeit indirect) of the rate at which 
customers were being restored, and the trend by early Friday afternoon was encouraging. The 
following graph was prepared and circulated at the SEOC as an indicator of the restoration rate. 


24 Figure provided by the U.S. Department of Energy, Office of Energy Assurance. 
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Load Rebuilding for Detroit Edison Customers 
8,800 MW Normal Estimate Load for Friday and Saturday, 
Aug. 15-16, 2003 - Available Capacity Approx. 7,500 MW 
8,000 -] 



5:00AM 11:30AM 1:30PM 2:10PM 3:30PM 6:00PM 7:00AM 9:30AM 1:43PM 


Second, reports from city and county emergency operation centers provided further information 
as areas were restored. Third, one of the early indications of the extent of the outage was a map 
that showed the state’s 800 MHz radio system towers that were operating on emergency backup 
generators. 


Section 4.3.2: Collecting Information 

As soon as the PSC Staff arrived at the SEOC, they began a systematic inquiry directed to the 
operations centers and emergency personnel at the affected utilities to ascertain the extent of the 
power outage and the status of a return to power. The U.S. Department of Homeland Security 
informed us that the blackout was not an act of terrorism. We also communicated with DOE and 
regional energy officials. After ruling out a terrorist act, the question quickly turned to the 
source and extent of the blackout. First reports on the early evening of Thursday, August 14, 
suggested a power plant was on fire in New York City, and then it was a transformer. Next, 
reports came in that lightning had hit a power plant operated by the Niagara Mohawk utility in 
western New York, problems in Canada were then suggested, and at one point ABC News 
reported that the problem originated in Michigan. None of these reports proved to be accurate, 
but each was checked to determine if the information could be verified. Later reports began to 
focus on transmission lines in northern Ohio that tripped off the system. Web site searches on 
news sites was one means used to track the reports that were coming in, sometimes second and 
third hand, and suffering the distortion that occurs as information is removed further and further 
from its source. 

NERC, located in Princeton, New Jersey, was an important source of information. A conference 
call held by NERC on Friday, August 1 5, provided a helpful update and a preliminary report of 
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events posted to the NERC web site was provided to the Governor on Saturday morning, August 
16. 


Section 4.3.3: Public Information Needs 

One of the most important crisis management actions state government takes during an 
emergency is to provide information to the public. Timely, accurate information about an 
emergency can help prevent confusion and uncertainty, as well as enlist the support and 
cooperation of the public. It is also important that the public understand what caused the 
emergency and what needs to be done to ease and eventually resolve it. The Michigan Energy 
Emergency Operations Manual provides a useful guide on how a public information program 
should be used and operated in an energy emergency. Following these guidelines, the PSC 
provided information to the Governor and the public regarding the status of the power outage, 
what steps were being taken to resolve the situation, and how the public could help. 

The PSC public information efforts started at the onset of the power outage. Early in the evening 
of August 14, two PSC Consumer Alerts were faxed to the Governor’s office from the SEOC. 
The alerts were entitled “Tips For Buying And Using A Portable Generator” and “Surviving 
Electrical Power Outages - What You Can Do If You Lose Your Electric Service.” This 
information was then modified to reflect the current outage conditions and posted to the 
Michigan.gov web site (see Appendices A-8 and A-9). 

Also on the first night of the power outage, we provided information to the Governor and her 
staff on the affected utilities’ power restoration efforts in preparation for her WKAR television 
broadcast appearance at 10:00 p.m. the night of August 14. 

In the days following the power outage, PSC Staff continued to provide information as needed 
on utility restoration efforts, reports on preliminary causes of the power outage, and other 
outage-related information to the Governor and the public. The Governor and the PSC Chair 
utilized this information during the press conference that was held at noon on August 15. Also 
on August 15, the PSC issued a press release asking Michigan citizens throughout the State to 
conserve electricity to reduce demand and help stabilize the system in light of the fact that a 
number of power plants were off line, both in and out of the affected area. 

At the request of the Governor’s press office, during and following the outage, the PSC 
responded to media inquiries from the trade press; national press (New York Times, Boston 
Globe, Washington Post, Reuters, Associated Press, Dow Jones, Wall Street Journal); and 
Michigan media (WJR, WWJ, WDET, Michigan Public Radio, Detroit News, Detroit Free Press, 
Oakland Press, Jackson Citizen Patriot, Booth newspapers; and Bay City Times). A number of 
these entities wanted specifics on the geographic areas affected by the power outage. In these 
cases, we directed them to the PSC web site where maps of the affected service territory areas 
could be found. 27 


27 http:7/www.cis.state.mi.us/nmse/electric/map.htm 
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After the power had been restored, the PSC continued providing information on the power 
outage. On September 3, 2003, the PSC Chair testified before the U. S. House Committee on 
Energy and Commerce on the power outage. 


Section 4.3.4: Declaration of Emergency 

On the morning of Augsut 1 5, Governor Granhoim declared a State of Emergency under the 
provisions of both the State’s Disaster Act and the Energy Emergency Act. 

With this declaration, two things needed for coping with the power outage and a possible 
gasoline shortage occurred: 1) the declaration allowed petroleum suppliers to import gasoline to 
make up for the dislocation of supply, and 2) a driver hour waiver provision automatically went 
into effect under contingencies established by the Federal Motor Carrier Safety Administration 
(FMCSA) that would allow the drivers of trucks carrying gasoline and other needed supplies into 
the Detroit area to drive for extended hours. 

The initial declaration was followed up on August 22 by a Declaration of Energy Emergency 
caused by the fact that the power outage had adversely affected eight refineries in the U.S. and 
Canada, including the Marathon refinery in Detroit. The loss of production at the damaged 
refineries posed the potential for a gasoline shortage for the Detroit Metro area, creating the 
potential for an energy emergency. A second Executive Order, also issued on August 22, 2003, 
continued the suspension of rules for gasoline vapor pressure because strict compliance with the 
rule could “...prevent, hinder, or delay necessary action...” to cope with a gasoline shortage. 

The basic authority for the Governor’s ability to issue the Executive Orders discussed above 
comes from two pieces of legislation. First, the State’s emergency management legislation 
(1976 PA 390) was enacted in December 1976 and amended in April 1990. This act replaced 
Michigan’s Civil Defense Act and broadened the scope of emergency management. Act 390 also 
brought the State into compliance with provisions of the Robert T. Stafford Disaster Relief and 
Emergency Assistance Act (P.L. 93-288, as amended), which provides federal assistance in 
declared (by the President) emergencies or major disaster situations. 

The Governor also acted under Section 1 0.82 of the Michigan Compiled Laws (MCL), 1 982 PA 
191, as amended. The purpose of this act is to allow for the declaration of a state of energy 
emergency, to provide for procedures to be followed after a declaration of a state of energy 
emergency, to create an energy advisory committee (EAC) to the governor and prescribe its 
powers and duties, to prescribe the powers and duties of the governor, to prescribe penalties, and 
to repeal certain acts and parts of acts. The EAC is chaired by the Chair of the MPSC, as 
provided for in Executive Order 1986-17. Under the provisions of Act 191, the Governor can 
declare an energy emergency on her own initiative or at the advice of the EAC. During the 
power outage and subsequent events involving the damaged refineries, the EAC Chair provided 
information to the Governor advising that an energy emergency was potentially imminent. The 
Governor subsequently declared an energy emergency. 
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Section 4.3.5: Contacts 

There were considerable contacts between the State and other agencies, both local and federal, 
DOE and the Department of Homeland Security (DHS), and other states. Staff members at the 
SEOC were in communication with both federal agencies and other states. Generally, 
communication worked, although there is always room for improvement. The evolving 
relationships and emergent roles of DOE and DHS are in need of clarification. The information 
from federal agencies in Washington was somewhat fragmented, and available information was 
not always made readily available. The DOE has been working with state officials to prepare a 
protocol that would enhance information exchanges and coordination between states and the 
federal government in an energy disruption, such as the power outage. This will require a better 
delineation of energy emergency responsibilities between DOE and DHS. Several DOE 
functions, including an office that had oversight of critical energy infrastructure, were transferred 
to DHS. 

The DOE Office of Energy Assurance (OEA) is working on a procedure for communication that 
can be used in any kind of energy disruption. This should allow for a more systematic sharing of 
information between the states and federal government to assure that information is rapidly 
distributed to key participants and to avoid misinterpretation of information. OEA played a 
major role in DOE’s response to the blackout, assisting state and local authorities and industry, 
and advising Energy Secretary Abraham. 


Section 4.3.6: Telephone System Operation 

In the early evening hours of August 14, it became apparent that there was an issue concerning 
fuel required for standby generators used by the local phone system. In addition, a number of 
government offices and private entities were operating generators in order to maintain power at 
their own sites and were competing for limited fuel supply. Although all had arrangements with 
fuel suppliers, some suppliers and distributors did not have the capability to pump the fuel from 
underground storage tanks without power or their own generators. 

Another problem that came up Friday night involved a critical site, which handles all 91 1 calls in 
Oakland County, which was running on standby generators. Because the generators were not 
fully capable of meeting cooling needs due to the hot weather at this site, additional generators 
were moved to this location when power was restored Friday evening. The generator at this site 
was relocated to other critical needs. There is now a plan to add additional generation at this 
location. 

The local phone system, which is operated by SBC, requested assistance from the SEOC in 
locating supplemental supplies of gasoline, kerosene, or #1 diesel fuel to assure the continued 
operation of the local telephone system. This fuel was needed for both standby generators and 
company vehicles to allow travel to remote locations to assure continued operation of equipment. 
During the power outage, more than 3 million customer lines in the central offices and 380,000 
customer lines serviced by remote terminals could have been impacted by the loss of commercial 
power. However, because of SBC's emergency backup procedures, only 50,000 customers were 
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impacted for a little over an hour. And this was despite the fact that the blackout generated an 
increase of 149 percent in the volume of calls placed on the SBC network. 

Initially, 120 central offices and 2,300 remote terminals lost commercial power throughout 
Lansing and southeast Michigan. However, by using backup generators and batteries, SBC was 
able to maintain service in the offices and to the remote terminals even though they were without 
power for an average of 28 hours. Only the Mt. Clemens north central office was impacted, and 
those customers could not call outside of their local area for 68 minutes during the early morning 
on Friday, August 15. 

As indicated, SBC called the SEOC for assistance and the PSC Staff at the SEOC identified a 
number of suppliers that had fuel available and could supply SBC's needs. Ultimately, SBC 
spent almost $650,000 purchasing 320,000 gallons of fuel. Some of that fuel was purchased 
from suppliers identified by the SEOC and some of it from existing suppliers. 

SBC owns two large trailer-mounted generators and a number of smaller portable generators; in 
addition, SBC borrowed two larger generators from a local manufacturer and moved these 
generators around to meet their needs. Each of the 2,300 remote terminals needed to have their 
batteries periodically recharged. SBC accomplished this by having its technicians move portable 
generators to a terminal, powering up the generator until the terminal batteries were folly 
charged, and then moving the generator to another terminal and repeating the process. This was 
an additional challenge because a number of terminals are in rural locations and not easily 
identifiable, particularly at night. 

No long distance companies reported any problems, in part, because their services are dependent 
on the operation of the local phone system. In addition, because many had developed business 
continuity plans; they were able to continue operations. Some cell phone companies did lose 
service for a time, and as the duration of the outage became more extended, they were also at the 
point of needing additional supplies of fuel for generators; however, the power was restored 
before this became a problem. 


Section 4.3.7: Communications 28 

There was foil and robust communication between the appropriate federal and State agencies, 
although further improvements can be made. DHS and the Federal Emergency Management 
Agency (FEMA) in DHS were in regular, consistent contact with the SEOC. DEQ, MPSC, and 
the National Guard were communicating with the EPA, DOE, and the National Guard Bureau, 
respectively. 

Two suggestions for improvement can be made, however. First, the reports given to both DHS 
and FEMA Region V were redundant. While the pace of the emergency response was such that 
this was not a serous problem, this redundancy should be eliminated as the reorganization of 
federal agencies within DHS is completed. Some material was sent over phone lines by 


28 Some of this information is taken from the Congressional Testimony of Col. McDaniel, Homeland Security 
Advisor to the Governor, to House Select Committee on Homeland Security, September 17, 2003. 
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facsimile, but e-mail with text-based documents would have been a better alternative, since the 
information can more readily be shared and incorporated within E-Team. Second, all 
communication was by telephone and, given the intermittent outages of commercial telephone 
service elsewhere in the State, a backup system needs to be instituted that is not reliant on 
commercial lines. There is a wireless system between FEMA Region V and the SEOC, and this 
capability could be expanded. 

The DOE Office of Energy Assurance (OEA) is working to enhance and expand the Energy 
Emergency Information Coordinators (EEIC) Program" with the assistance of the National 
Association of State Energy Officials and the National Association of Regulatory Utility 
Commissioners. This program provides a point of contact in each state for information on 
energy supply, availability, and potential distribution problems. This set of contacts will be 
expanded to include contacts in public utility commissions on matters related to electric and gas 
supplies. While the vehicle for communication is principally based on non-secure e-mail at 
present, other communication technologies will be examined as part of this effort. It is intended 
to provide for an additional communication channel between the states and federal government 
to provide for the exchange of information between and among states. These contacts provided 
for some of the communication that occurred during the power outage. To be successful, this 
system will need broad-based participation by the states, and information to be shared must be 
reliable, timely and useful. 

Internal communications, both within a State agency and between employees of the State and 
local agencies, worked very well. Over the last 12 years, the State of Michigan has spent in 
excess of $220 million to create a statewide 800 MHz digital trunk radio system. This system 
provides full interoperability for all organizations using it. Currently, 374 different public 
agencies use the Michigan Public Safety Communications System as their primary radio 
communication, and another 90 agencies use the system for emergency management purposes 
only. The member agencies include all state agencies, as well as counties, townships, tribes, and 
federal agencies (the FBI, U.S. Customs, Bureau of ATF, and Forest Service). There are 
currently more than 1 1,000 radios on the system. 

There were no interruptions to the system anywhere during the blackout because the control 
center and all antennae have independent generators. Four of the five affected counties, as well 
as many municipalities within those counties in the declared emergency area, are now 
considering joining the Michigan Public Safety Communications System. 


Section 4.3.8: Cyber Systems 

Cyber systems were shut down in areas affected by the power outage unless they had backup 
batteries or generation. Many computers and nearly all servers supporting large-scale systems 
operate with Uninterrupted Power Supply (UPS) units. The UPS provides power conditioning 
(controls fluctuations in current and voltage which can damage circuitry), and when power is lost 
automatically shifts over to battery backup and then directs the server to a controlled shutdown 


29 http://www.naseo.org/eeic/default-htm 
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to avoid errors in software operations. When power is resumed, the servers then reboot, as 
normal, bringing operation systems back on line. In not all instances will this execute properly 
and some servers will require a technician to bring the system back on line. In some cases the 
age of equipment may have been a factor in returning the system to operations. 

Computer operations, which run multiple servers, also require cooling in the server room. Since 
the outage occurred on a hot summer day, it was necessary to get air conditioning systems back 
into operation, allowing the server room to be cooled to normal operating temperatures before 
the servers could be returned to service if back up generation was not available. 

A significant challenge faced by computer system operators was that the outage occurred in the 
middle of an effort against the “MSBLASTER” worm virus attack and the “SoBig” virus. The 
system attacks as a result of these two viruses had already caused some disruptions to computer 
systems and the power outage compounded the problems and the response. 

Many of the State of Michigan’s computer operations were affected. Most Lansing and 
southeast Michigan computer hardware encountered a hard shutdown, with the exception of most 
voice systems. The Department of Information Technology (DIT) Command Center was 
without power in the Hannah Building in downtown Lansing. 

Actions taken by DIT included a quick response to assure most servers in data centers were 
powered down prior to exhaustion of UPS battery power to eliminate problems when power was 
restored. The DIT Emergency Management Coordinator went to SEOC at 7: 1 5 p.m. on 
Thursday, August 14, and a DIT secondary command center was activated at the State Secondary 
Complex, which still had power. Additional fuel to power these generators was not available 
when planned due to the heavy demand on fuel suppliers, and these server operations were cut 
over to power from the grid at about the time fuel supplies for the generator were exhausted. 

Additionally, the following actions were taken: 

• Established an ongoing DIT conference call for problem resolution and status 
updates. 

• Identified all critical customers facing processes, i.e., Food Stamps, UA checks, 

CSES batch, Medicaid provider batch run, etc. 

• Performed modification so Governor could update Michigan.gov Web site. 

• Identified status of existing second and third shift staff and established a plan of 
action for scheduling critical work. 

• Worked with Department of Management and Budget (DMB) to restore failed air 
conditioning units. 

• When power returned, waited until data centers had cooled and followed start up 
procedures in all buildings. 

• Resolved network and hardware issues caused by failed components on Friday, 
August 15, in Lansing area and through Monday in Detroit. 

One of the lessons learned in the computer system operations was that any disaster recovery 
planning that had been done previously proved very helpful in the recovery of systems. For 
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many, the procedures that came out of the preparation for the rollover to the year 2000 (Y2K) 
helped immensely and exercises held since then were also very useful. The use of toll free 
conference calls proved to be an invaluable communication tool, as cell phones could not always 
be relied on. It was also clear that contact lists need to delve deeper into the organization and 
need to be listed on paper. 


Section 4.3.9: Water and Wastewater Systems 30 

Detroit operates the largest water and wastewater utility in Michigan and the third largest in the 
U.S. It serves 4.3 million people in 126 communities in nine counties in southeast Michigan. 
Highly trained and certified professionals operate well-maintained water and wastewater 
treatment facilities. The drinking water facilities consist of five water treatment plants, 22 
booster-pumping stations, and over 13,000 miles of water transmission and distribution mains. 
The system also consists of telemetry and automated SCADA equipment. The drinking water 
produced at the water treatment plants and pumped throughout the distribution system is 
monitored in accordance with Michigan DEQ requirements for chemical, microbiological and 
radiological contaminants. 

Detroit lost power to all pumping stations at the five water treatment plants, as well as system 
control telemetry. Of these five plants, Springwells, Southwest and Lake Huron used backup 
generators to restart within hours. Several communities called the system’s control center to 
complain of low or no pressure; however, many communities were able to maintain reduced 
pressure due to location and storage capacity. One Detroit Edison power feed was restored to 
the Lake Huron water treatment plant at approximately 7:30 p.m. on August 14. The three plants 
combined supplied approximately 430 million gallons per day (mgd) of drinking water to the 
distribution system during the blackout. Six (of 22) booster-pumping stations have backup 
generators, but only four stations were functional during the blackout. The annual average water 
demand for the Detroit Water and Sewerage Department (Detroit WSD) system is approximately 
600 mgd. With prompt implementation of water restrictions during the blackout, 430 mgd of 
water should have been sufficient to maintain pressure in the distribution system. 

A “boil water” advisory was issued for the entire Detroit WSD service area at 7: 15 p.m. on 
August 14 and rescinded at 3 p.m. on August 1 8. The boil water advisory was necessary because 
of a system-wide pressure loss that resulted from the blackout. The large number of residents 
and communities affected by the water emergency, the high level of involvement by state and 
local officials, and the potential health implications made this a serious matter of concern. 

Rule R 325.1 1206 of the Michigan Safe Drinking Water Act (PA 399 of 1976 and 
Administrative Rules) reads, “For a type I public water supply, a means shall be provided to 
continuously supply finished water to the entire distribution system during periods when the 
normal power service is interrupted.” This rule was implemented by requiring each water 
treatment plant to have a minimum of two electrical feeds from two separate substations, backup 


30 Much of the information in this section is taken from A Report on the Detroit Water and Sewerage Department 
System during the Blackout of 2003 , August 2003, prepared by the Michigan Department of Environmental Quality, 
Water Division, Field Operations Section, Southeast Michigan District Office, 
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generators, or sufficient gravity storage. The Detroit system exceeded this requirement at all 
water treatment plants and six booster stations. Each of the Detroit water treatment plants and 
booster-pumping stations has two electrical feeds from two separate substations, with the 
exception of Waterworks Park and Northeast, which has three separate feeds from three separate 
substations. Springwells, Lake Huron, and Southwest water treatment plants have backup 
generators able to deliver approximately 430 mgd total (near average day) to the distribution 
system. Six booster-pumping stations have backup generators as well. 

The separate electrical feeds provide for a contingency more frequently seen with storm outages 
where some localized areas are affected and others are not. This outage was highly unusual in 
that the entire electrical system failed and the power companies were required to initiate black 
start, procedures. Under these procedures the process begins at the power plant and restores the 
power moving from the plant outwards. Restoration of service to water treatment plants is an 
important priority in the overall electrical power restoration effort. 

During the August 14 power outage, wastewater utility professionals attempted to minimize 
negative impact from the sanitary sewer system. All wastewater systems are unique, and it is up 
to the licensed wastewater professionals to handle emergencies to affect the least damage and 
best outcome based on the capabilities of their systems. 

In addition to some wastewater systems having emergency backup electrical generators, some 
wastewater systems without backup power were able to store sanitary sewage in the gravity 
collection system. However, after filling the space in the limited storage of the sanitary sewers, 
utility personnel were faced with options of either allowing raw sewage to back up into 
residential basements, with significant public health issues, or overflowing at some point in the 
system to the receiving waters - a lake or a river. Overflows are typically managed to allow for 
partial treatment followed by disinfection as much as practicable. 


Section 4.3.10: Driver Hour Waivers 

The limits on the number of hours truck drivers can normally be on the road were suspended to 
assist with the recovery and re-supply following the power outage. The Federal Motor Carrier 
Safety Regulations (FMCSR) (390.23) provide relief from compliance with most safety 
regulations when an emergency is declared. This means a carrier would have to comply with 
Controlled Drug and Alcohol Testing (382) and with CDL Requirements (383), but not the 
provisions dealing with the Driver Qualifications (391), Hours of Service (395) and Maintenance 
of Vehicles (396). For example, a carrier would be granted relief from complying while 
providing assistance to the emergency. According to 49 CFR 390.5, an “Emergency means 
any... storm (e.g., thunderstorm, snowstorm, ice storm...) earthquake... explosion, power outage, 
or other occurrence, natural or man made, that interrupts the delivery of essential services... or 
supplies (such as food and fuel) or otherwise immediately threatens human life or public 
welfare. ...” Under such situations, the FMCSA Field Administrator may declare emergencies if 
there is a regional crisis, which justifies such regulatory relief or when an emergency has been 
declared by the President of the United States, the governor of a state, or their authorized 
representatives having authority to declare emergencies. 


80 



457 


Under a declaration of emergency under CFR Part 390, governors have broad authority to 
respond to disastrous situations. This authority in nearly all cases would automatically give 
effect to driver hour waivers upon an emergency declaration by a governor. FMCSA recognized 
a governor’s authority as the basis for granting interstate waivers for drivers supplying the 
affected areas in a state in relation to disaster situations. Therefore, once Governor Granholm 
issued the emergency declaration at 9:15 a.m. on August 15, 2003, the waivers automatically 
went into effect. This waiver helped assure the delivery of needed supplies to southeast 
Michigan, including water and later gasoline, and remained in effect until the emergency 
declaration was rescinded. Notification was sent out over the State Police Law Enforcement 
Network and e-mails were sent to the energy and transportation trade associations informing 
them of this action. While these waivers should not be employed any longer than needed by 
companies, they technically remain in effect during the period under which the emergency 
declaration is in effect. The energy emergency declaration following the power outage was 
rescinded on September 30, 2003. 


Section 4.3.10: Detroit Marathon Refinery 31 

Late on August 14, 2003, reports to the SEOC incorrectly indicated that there was a fire at the 
Marathon refinery located in southeast Detroit and storage tanks were a concern. What in fact 
occurred was that when the power went out, the refinery went into emergency shutdown 
procedures. These procedures provided in part for petroleum products being processed under 
pressure to be dumped to safety flares. The flares, a safety valve for the refinery, burned off 
petroleum products in process, producing a flame that was reported to be anywhere from 30 to 
75 feet high. Over a darkened city, this was very prominent. In addition, one of the units, a 
carbon monoxide boiler, did not shut down properly, causing an explosion, resulting in the 
release of a mixture of hydrocarbons and steam. The reduction in water pressure also was a 
compounding problem in the shut down of the refinery. 

The release of the hydrocarbons and steam created concerns as to whether or not these emissions 
were toxic. Testing done at the refinery suggested that it was not an immediate hazard and the 
presence of benzene or hydrogen sulfide was ruled out. However, as a precautionary measure, 
the immediate surrounding residential communities located in southeast Detroit and Melvindale 
were evacuated. Additionally, 1-75, which runs immediately adjacent to the refinery, was 
temporarily shut down to facilitate with the evacuation. The hydrocarbon release was contained 
at the Marathon refinery by about 8:00 a.m. the morning of August 15, and 1-75 was opened back 
up to traffic. However, people that had been evacuated to shelters were not allowed to return to 
their homes until later in the day on August 15, following additional testing by the EPA, which 
had come into the area to monitor the air quality effects. Once it was determined that those 
effects were not harmful, residents were allowed to return to their homes. 

The Marathon refinery can process 74,000 barrels of crude oil per day into a variety of petroleum 
products. Approximately half of the production from the refinery is gasoline. The gasoline 
being produced at the Marathon refinery at the time of the power outage was of a specification 

31 Celeste Bennett, Michigan Department of Agriculture, provided some of the information in this section. 
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designed to meet air quality requirements in southeast Michigan. The Motor Fuels Quality Act, 
P.A. 44 of 1984, as amended, requires that Wayne, Oakland, Macomb, Washtenaw, Livingston, 
Monroe, and St. Clair Counties use lower evaporating gasoline (7.8 pounds per square inch (psi) 
Reid Vapor Pressure 32 (RVP) or less) from June 1 to September 15 of each year. 33 This 
particular specification is intended to reduce evaporative emissions of volatile organic 
compounds (VOC) from gasoline during the summer months. This program is part of the State 
Implementation Plan (SIP) submitted to the EPA in 1996 in response to the clean air act 
mandates. The balance of the state uses RVP 9 psi gasoline throughout the summer and the same 
RVP limits are in effect statewide for the remainder of the year. The State’s failure to enforce 
the 7.8 psi gasoline requirements could result in federal sanctions of highway dollars or 
implementation of federal enforcement measures. 

Due to the damage at the refinery, it did not go back into full production until August 23, eight 
days after the onset of the outage. This meant that during that time, the refinery did not produce 
approximately 500,000 barrels of petroleum products, of which about half was gasoline with the 
7.8 psi specification required for southeast Michigan 

Marathon alone supplies an estimated 38 percent of the 7.8 psi gasoline used in that area by 
utilizing full refinery capacity all summer long and bringing in additional 7.8 psi product to meet 
demand. And, although the Toledo and Sarnia, Ontario refineries were only offline for a day or 
two, the lost production from these production facilities also contributed to a shortfall in supply 
of the 7.8 psi gasoline. According to the Energy Information Administration, the Marathon 
refinery and the two refineries in Toledo typically supply about 6-7 percent of distillate fuel oil 
(heating oil and diesel) and about 7-8 percent of gasoline demand in the Midwest and are even 
more important to the upper regions of the Midwest, where they are the major suppliers to 
Michigan and parts of Ohio. Each day these refineries are down, they do not produce 8 million 
gallons of gasoline, which is about two-thirds of what the entire State of Michigan uses in one 
day. 


Section 4.3.11: Gasoline Distribution Problems 

As a result of the disruption to refinery production both in Detroit and Toledo Ohio, the available 
supplies of 7.8 psi gasoline were quickly depleted. Only about 9 percent of the stations in the 
Detroit area were operational during the power outage and they reported customers were lined up 
in the street filling every canister they could. Stations were not able to replenish their supply of 
7.8 psi gasoline because all but one Detroit terminal with 7.8 psi gasoline were without power. 
The one terminal with power was supplying petroleum products, but would not send their own 
tanker trucks out because of unsafe road conditions due to the lack of operating traffic signals. 
Some stations with available gasoline shut down because they were unable to handle consumers’ 
behavior and had concerns for employee safety. Police were dispatched for crowd and traffic 


33 “Reid vapor pressure” means the absolute vapor pressure of volatile crude oil and volatile non-viscous petroleum 
liquids, except liquefied petroleum gases, as determined by A.S.T.M. D-323-72. 

33 After September 15, gasoline RVP requirements for the Detroit area are as follows: September 16 to October 31 - 
13.5 psi; November 1 to March 31 - 15 psi; April 1 to April 31-13.5 psi. Ethanol blended fuels are 1 psi higher. 
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control at numerous stations in and out of the Detroit area as lines of cars backed up for as much 
as two hours. 

Consumers began to migrate west and north in search of available gasoline and operational 
stations in Pickney, Manchester, and Chelsea reported a run on gasoline. Stations in outlying 
areas began reporting that they were out or running out of gasoline from the deluge of customers 
“stocking up.” Reports were received of consumers filling as many as 10 portable gas cans at a 
single purchase. There were reports that at some gas stations tanker trucks could not maneuver 
through the heavy traffic to get in to restock the stations, even with 9.0 psi gasoline, so some 
stations were forced to shut down until consumer demand coming out of Detroit had slowed 
down. 

Recognizing the potential supply shortfall of 7.8 psi gasoline in southeastern Michigan, the 
Governor declared a State of Emergency and a State of Energy Emergency on Friday morning, 
August 15. At approximately 9:15 a.m., Executive Order No. 2003-10 was issued by the 
Governor to suspend the environmental specifications for gasoline used in southeast Michigan 
counties to ensure that the area stations could receive any available gasoline. The DEQ 
requested that the EPA exercise enforcement discretion in regards to the SIP 7.8 psi requirements 
in southeast Michigan. The request was granted and was in effect until midnight on Friday, 
August 22. Notification of the suspension of the 7.8 psi requirements was sent out to all 
Michigan gasoline trade associations. Notice was also sent to Oil Price Information Service, 
which is utilized heavily by the petroleum industry for up-to-date news. 

The following Wednesday, August 24, EAC Chair Lark advised the Governor of an impending 
energy emergency due to dwindling gasoline supplies as a result of the continued shut down of 
the Marathon refinery in Detroit. At the request of the DEQ, the EPA extended the state’s 
enforcement discretion for southeast Michigan until September 3. On Thursday, August 21, 
Executive Order No. 2003-1 1 was issued by the Governor, which rescinded the State of 
Emergency and which also declared that an energy emergency existed for the state of Michigan 
due to the loss of gasoline supplies. That same day, Executive Order No. 2003-12 was issued by 
the Governor, which continued the suspension of environmental specifications for gasoline 
required for use in southeastern Michigan. 

Inspectors from the Michigan Department of Agriculture, Motor Fuel Quality and Weights and 
Measures Division, who enforce the specification requirements for the State Implementation 
Plan under the air quality requirements, determined that a number of stations in southeastern 
Michigan had taken advantage of the waiver of the requirements and had blended a 9 psi 
gasoline into 7.8 psi gasoline already in tanks in retail gas locations in the Detroit and southeast 
metropolitan area. Based upon this information, it became apparent that it would be impractical 
to drain the tanks of this blended gasoline so that 7.8 psi could be put in place. A complete 
change over of gasoline stocks from 9.0 psi to 7.8 psi normally requires about a one-month lead 
time. Once the 9.0 psi product was in the distribution chain, it would continue to impact upon 
gasoline supplies for several weeks. 

By Saturday, August 16, power was restored to much of southeastern Michigan and consumers’ 
gasoline buying habits returned to normal by that evening. Power was restored to the Marathon 
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and Toledo refinery operations, but production could not be resumed until after a total systems 
check and damage assessments were completed. Sunoco’s Toledo refinery was back on line 
August 17, but Marathon was unable to return to full production until August 23. Other major 
marketers to southeastern Michigan indicated they had sufficient 7.8 psi gasoline to meet their 
normal market share, but did not have surplus product to make up for the refinery’s shortfalls. 

To resolve the shortfall of the availability of the 7.8 psi gas, the DEQ had originally requested 
that EPA permit the State enforcement discretion until September 5 to allow sufficient gasoline 
supplies for the holiday weekend. The EPA granted the state enforcement discretion until 
September 3, at which time a re-evaluation would be done. However, on August 28, because of 
the commingling tank contamination and the continued potential for problems with obtaining 
sufficient quantities of 7.8 psi gas, the DEQ requested that the EPA extend the environmental 
waivers until September 15, at which point the summer specifications would no longer be in 
effect. The EPA granted this discretion for distribution, transportation and sales until September 
1 5, provided regulated parties took all reasonable steps to produce and supply 7.8 psi gasoline to 
southeast Michigan. 

During this same time, beginning on August 1 5, a surge in demand for gasoline took place as 
people in the areas affected by the power outage drove to locations outside the affected area to 
fill up their vehicles. This resulted in a substantial drain of gasoline supplies around the 
periphery of the power outage area. In the week that followed the power outage, gasoline prices 
in both Michigan and the Nation rose at record rates. The average price in the Detroit area prior 
to the outage was $ 1.59 per gallon 34 on August 1 1, 2003. By August 25, the price had increased 
to an average of $1.77, and just prior to the Labor Day weekend peaked at nearly $1.88 per 
gallon at many stations. These price increases were not unique to Michigan and, in fact, these 
increases were seen nationwide. 

The reason for the increase in prices seemed to be the result of a combination of events, which 
included rising crude oil prices through the month of August and a significant increase in 
gasoline demand that was evidenced by sharp draw downs in gasoline inventories. Even though 
the loss of refinery production was relatively minor, it nevertheless contributed to a tightening of 
supply, which further accelerated the price increases. As a comparison, problems with a gasoline 
pipeline near Phoenix, A Z had caused gasoline prices to go to $4.00 a gallon at some locations 
for a temporary period of time. Another one of the reasons for the accelerated demand for 
gasoline was the cooler than normal weather during the months of June and July, which tended 
to defer travel until August, at which point improved weather patterns caused people to take to 
the roads in notable numbers. Gasoline prices typically show an increase leading up to the Labor 
Day weekend and a seasonal decline following Labor Day weekend. This trend has taken place 
as gasoline prices have now settled back down along with the decline in crude oil prices. 


34 http://www.autoclubgroup.CQm/michigan/autos/fuel eauee.asp 
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Section 4.3.12: Food Supply 35 

The most significant events impacting the food supply were the loss of electricity to power 
refrigeration and the boiled water advisory that resulted from the loss of municipal water 
pressure due to the loss of electricity to pumping stations. Large grocery retail chains have 
contingency plans in place to provide auxiliary refrigeration to impacted stores. Smaller grocery 
retail establishments and restaurants faced significant challenges to prevent their perishables 
from becoming compromised. 

The Michigan Department of Agriculture’s response included directing extensive food safety 
inspection resources into the area to inspect grocery retail establishments, guidance to local 
health departments regarding restaurants, and extensive public outreach. 

The food and agriculture infrastructure experienced the same difficulties in communications as 
other sectors. The dependence on cellular and cordless telephones, as well as the computer 
assisted switching in both the private sector and government made communications difficult. ■ 

The lack of the availability of commercial sources of gasoline posed an impediment to 
responders being sent into the impacted area. Allowing Michigan Department of Agriculture 
inspectors to fuel at State Police Posts solved this problem. 

Resources that would be available in many situations to transport potable water, bulk milk 
haulers, were busy transporting milk from areas impacted by the loss of electrical power. 

The amount of food being discarded by grocery stores posed a public health hazard as an 
enticement to poor people and an attraction to rodents. Initially, licensed waste haulers were not 
willing to deviate from their pre-existing contract obligations and landfills were not readily 
available on the weekend. The loss of electricity created a situation where public health was 
potentially impacted by the availability of a landfill on a Sunday. 


Section 4.3.13: Transportation 36 

The transportation system was also affected by the power outage in the following ways: 

• Traffic signals were not functioning which meant, in some cases at busier intersections, 
police personnel had to be deployed in directing traffic when their services could perhaps 
have been needed elsewhere. 

• Pumps used to keep depressed highways from flooding had to be powered with portable 
generators. This required crews to work 24 hours a day until power was restored. Some 
areas received so much rain that the crews could not keep up and some highways (1-94 in St. 
Clair Shores, M-39 in Detroit) were temporarily flooded. 


35 Prepared by Robert Tarrent, Emergency Management Coordinator, Michigan Department of Agriculture. 

36 Information in this section was provided by Eileen Phifer, Michigan Department of Transportation, and, in part, 
was taken from Congressional Testimony of Col. McDaniel, Homeland Security Advisor to the Governor. 
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• Systems at the Michigan Intelligent Transportation System Center were without power. 
Video cameras, changeable message boards, and the Center's Web site all went offline. 
Communications with freeway courtesy patrols, outside media, and Michigan Department of 
Transportation was nearly impossible until the power was restored. 

• The Detroit Windsor Tunnel was shut down due to the inability to operate ventilation, but the 
Ambassador Bridge continued to operate. 

• The Ambassador Bridge in Detroit, the busiest commercial land port in the U.S., with 1 6,000 
tractor-trailers crossing daily, was also affected. Interestingly, both the bridge and U.S. 
Customs had their computers interrupted only momentarily until their backup systems 
activated. Canadian Customs, however, lost their computer data link, and thus their ability to 
verify trucking manifests electronically. As a result they were forced to visually and 
manually inspect the manifests and, if warranted, the freight itself. This resulted in an 
approximately four-mile back up of traffic for almost 24 hours on the U.S. side. 

• Metropolitan Detroit Airport was closed and all flights canceled until midnight on August 14. 
Flooding of the approach roads to the McNamara Terminal caused by a storm on August 15 
cut off vehicle access to the terminal temporarily. 


Section 4.4: Lessons Learned and Recommendations 

Even though the State as a whole was well prepared for the events following the August 
blackout, events such as this serve as a reminder that we can be better prepared. Most of the 
following items are in areas where work has been done that greatly contributed to the ability to 
respond to problems caused by the outage, but some gaps in planning and preparedness were 
evident and should be addressed. In general, improvement efforts need to focus on planning, 
assessment, communication, and training. 

1. Examine the current Emergency Electrical Procedures adopted in Commission Case No. U- 
4128 to determine if they remain valid. These procedures were last updated in 1979 and 
the electric utility industry has changed dramatically since then. 

2. Update Emergency Gas Procedures manual. Natural gas curtailment procedures have been 
adopted by the PSC for each jurisdictional gas utility in the State. However, many of these 
procedures may no longer be effective because large volume customers no longer buy 
directly from the local distribution utility. 

3. Update the PSC Energy Emergency Operations manual. This manual was last revised in 
May 1992. 

4. Update the Department’s Emergency Management Coordinator responsibilities as 
contained in the State Emergency Management Plan currently undergoing revision. 

5. Update contact lists and emergency procedures. While the contact lists in existence at the 
time of the power outage proved invaluable, some deficiencies in the lists were revealed. It 
is important for PSC Staff to maintain a professional working relationship with the key 
emergency contacts. Both the emergency contact lists and the communications procedures 
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have been updated since August 14, 2003. However, in the future, contact lists of 
emergency numbers - work, home and cell/pager - and e-mail addresses need to be kept up 
to date. This should be done on a regular schedule, preferably annually, and should include 
the following groups: 

A. State agencies 

B. Industry/private sector and non-profits 

C. Federal agencies 

6. Provide for additional PSC Staff training. PSC Staff that worked at the SEOC during the 
outage included individuals that were selected due to their availability, not necessarily 
their expertise in energy emergency procedures. A total of six individuals provided 
support at the SEOC. Additionally, a number of PSC individuals from public 
information, Energy Operations Division, and Energy Data & Security Section provided 
support to those at the SEOC from the Commission offices on Friday, August 15, 2003. 
While Staff at both the SEOC and the Commission offices provided invaluable help and 
support, previous training in the workings of the SEOC and energy emergency 
procedures would have been helpful. It is suggested that PSC Staff should be pre- 
designated to serve this function in case of future energy emergencies. Staff should be 
trained in the procedures of both the SECO and energy emergency procedures and should 
participate in annual emergency preparedness exercises conducted at the SEOC and 
elsewhere. The SEOC training should include training in the use of E-Team, 37 crises 
management software, and maintaining a duty log. 

7. Training for the Department’s Emergency Management Coordinator and alternates 
should also be provided on the use of E-Team and SEOC procedures. The list of 
emergency contacts in each bureau should be kept up to date and these bureau contact 
individuals should meet annually to review and discuss the Department’s emergency 
response procedures. 

8. Events should be more fully documented. Staff assigned to the SEOC should be trained 
in the use of E-Team, and events should be documented as they occur in E-Team. This 
will mean becoming more proficient in the use of the E-Team software and having better 
rules as to how this system should be used. Additionally, it is important that both a duty 
log and a phone contact log be maintained at the SEOC and by PSC backup support from 
the Commission offices. The PSC Staff assigned to do this support should be familiar 
with these procedures and trained to keep these records. This information is valuable 
both for keeping track of the event at the time and for assessing the emergency after it has 
been resolved. 

9. Evaluate interdependencies. Staff working on energy emergencies should be versed in 
the potential interdependencies involved in such an emergency. For instance, in this 
power outage, it quickly became apparent that the loss of electrical power seriously 
affected the water systems of many of the affected communities. This loss had the 


37 E-Team is an incident management software package recently implemented at the SEOC. It includes a data base 
that provides for tracking of events, allocation of resources, and information sharing. 
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potential to cause very serious problems had the power outage lasted longer than it did. 
These problems could have included lack of clean drinking water for public consumption, 
lack of adequate wastewater treatment abilities, and lack of water for firefighting 
purposes. 

Another interdependency that manifested itself as the power outage progressed was the 
need for fiiel for the generators many entities were running to maintain services 
(hospitals, telecommunication systems, etc.). While some institutions and businesses had 
the foresight to have generators in place, they had neglected to contract for fuel deliveries 
to continue running the generators for an extended period. In some instances, generators 
fueled by natural gas should be considered. 

Staff needs to be familiar with at least the concept of interdependencies and some of the 
more likely problems to spring from a power outage and related systems. Staff should be 
versed in potential solutions to some of these problems. For instance, in the case of 
generators running short of fuel, Staff made calls to area terminals and fuel supply depots 
in an effort to locate a supply. And, while the solution to interdependency may not lie 
with MPSC Staff (e.g., with the power outage, traffic signals were not working and police 
had to be dispatched to busier intersections to direct traffic), they should at least be aware 
of potential problems so as to advise the appropriate agency. 

10. The roles and responsibilities of state and federal agencies and associations need to be 
clearly delineated, including the U.S. Department of Energy Office of Energy Assurance, 
U.S. Department of Homeland Security, the PSC, National Association of Utility 
Regulatory Commissioners, and National Association of State Energy Officials. 

1 1 . Good up-to-date reference information should be provided. There is a need to have 
reference materials available on systems and infrastructure that may be involved or 
affected by an energy emergency. While PSC Staff had a great deal of this information 
available when it was needed (Emergency Operations Manual, State rules and regulations 
governing energy emergencies, contact lists, and statistics readily available regarding the 
Marathon refinery), some of this information was not readily available because it was 
contained in an electronic database on computers at the PSC offices. Since the PSC 
offices were without power, this information was not available. It is important that 
detailed information on all critical energy infrastructure and any other reference material 
deemed to be useful be developed and maintained in both electronic and hard copy 
formats, which will ensure that it is available when needed. 


Section 4.5: Conclusion 

In conclusion, the efforts of the PSC, its Staff, and other State and federal agency personnel 
made a significant contribution to avoiding much more serious consequences that might have 
occurred absent emergency management intervention. Statewide coordination of what turned out 
to be a widespread regional problem was essential to the maintenance of order and the 
restoration of public services. Finally, the tireless endeavors of utility personnel and 
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management as well as local first responders were critical in restoring essential services and 
maintaining public safety throughout the duration of the blackout of 2003. Nonetheless, this 
report reveals that a number of lessons were learned from this review, and a number of steps 
should be taken to improve the response to future emergencies. 
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PART V 

CONCLUSIONS AND RECOMMENDATIONS 


This part summarizes the most important conclusions and recommendations of this report. It is 
intended only as a convenient summary of items discussed more fully in the text. The most 
significant conclusions and recommendations in each sector are included: 


Section 5.1: Transmission 

• Michigan utilities and transmission companies were not the cause of the blackout. 
All of the events in the two and one-half hours preceding the power surges that 
occurred at 4:09 p.m. involved the facilities of FirstEnergy or American Electric 
Power in Ohio. No Michigan utilities or transmission companies were involved 
in these events. Information involving these events was not shared with Michigan 
companies prior to the blackout. 

• Congress must provide the FERC with the authority and responsibility to ensure 
(1) that mandatory reliability standards are in place, (2) that they are enforceable, 
and (3) that they include penalties for noncompliance. 

• FERC should permit but one regional transmission organization to operate in the 
Midwest market; but if it chooses to permit the existence of more than one 
organization, at an absolute minimum, there must be mandatory, enforceable rules 
that address issues that arise at the seams between the organizations. 


Section 5.2: Electric Utilities 

• Detroit Edison should conduct an analysis of the “In Service Application” system 
to consider modifications or alternatives that would function more effectively in 
the event of a similar blackout, and should report the results of its analysis to the 
Commission. 

• Detroit Edison should conduct an engineering analysis of the operation of 
its rupture disks to determine if any modifications are warranted and 
report the results of its analysis to the Commission. 
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Section 5.3: Emergency Response 

• There should be a review of the Emergency Electrical Procedures adopted 
in Case No. U-4128, which have not been updated since 1979. 

• The Public Service Commission should designate in advance the Staff 
assigned to the State Emergency Operations Center and provide training 
for that assignment. 
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Appendix A-l 

Governor's Statement to the People of Michigan 


Aug, 15, 2003 

The following is Ihe text of Governor Jennifer M. Granholm's statement to Michigan residents 
regarding the power outage, which was broadcast via satellite on Thursday evening: 

Let me begin by reiterating what you all heard the President of the United States say just an 
hour ago. The electrical outage that we are experiencing here in Michigan is not the result of a 
terrorist attack, but appears to be the result of some other natural occurrence - that caused an 
outage at a power plant in New York State earlier this afternoon. That outage in New York 
rippled across the Eastern United States and Canada, actually stopping here in Michigan. 

At this hour, utility crews are working to restore power in the affected areas. As you know by 
now, the outage affects mostly residents in the middle and southeast parts of the state. Detroit 
Edison serves 2.1 million customers in these areas, all of which are still out of power as I 
speak. In addition, 100,000 Consumers Energy customers are without power. 

Detroit Edison is saying that they are beginning to power-up their plants and they will 
continually be bringing customers back online. The utility cannot confirm exactly when all 
power will be fully restored, but they are hopeful that most customers will be back online before 
the end of the weekend. This will be a gradual restoration, but I am pleased to report that 
thanks to the swift response of utility crews and the power conservation of our citizens, reports 
are beginning to trickle-in that power is slowly coming back on in some locations. 

Oakland, Macomb and Wayne Counties have declared LOCAL states of emergency. At 8:30 
p.m. we fully activated the state's Emergency Management Operations Center, which allows us 
to have a central point of communication between the all state, local and federal agencies. 

Some people may have questions about steps they can take to remain safe and protect their 
families. We urge citizens, first and foremost, to remain calm. Also, for your safety, try to stay 
off the roads. If you must drive, treat all intersections as four-way stops. 

Beyond these important first steps, we encourage citizens to take the same basic precautions 
that you would in any other power outage situation. 

Unplug your appliances and major electronics - like computers, for instance. When power 
comes back on there may a surge which could damage these products. 

Importantly, while we encourage people to stay hydrated to stay cool, citizens should take 
steps to conserve water. Water is pumped to your faucet through pumps, which, of course, 
use electricity. 

As the evening and tomorrow progresses, we expect more communities to come back on-line. 

I want to thank ail of the utility workers and emergency personnel who have been working so 
hard to restore power. 

Finally, this is truly one of the instances where we are all in this together. So please be calm, 
be supportive of your neighbor, and take those extra precautions. Thank you. 
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JENNIFER M. ©RANHQUM 

GOVIWNOB 



State of Michigan 
OFFICE OF THE GOVERNOR 

LANSING 


Appendix A-2 


JOHN D. CHERRY, JR 

IT. GOvSftHOR 


EXECUTIVE PROCLAMATION 
STATE OF EMERGENCY 

WHEREAS, beginning on August 14, 2003, significant portions of the State of Michigan 
are experiencing the effects of a severe power outage, resulting in the loss of electrical power for 
countless Michigan residents, communities, and businesses, causing serious hardship for the 
citizens of the State of Michigan; 

WHEREAS, this power outage has resulted in the loss of power in numerous other states 
in the region, including Connecticut, Michigan, New Jersey, Ohio, Vermont, and the province of 
Ontario; 

WHEREAS, the power disruption has impaired and or threatens to impair the 
maintenance of essential public services, and therefore constitutes a danger to the health, safety, 
and welfare of the general public; 

WHEREAS, this event was caused by a utility failure and represents a threat of 
widespread or severe damage, injury, or loss of life or property; 

WHEREAS, it is in the best interests of the State of Michigan and its citizens that 
appropriate measures be taken to assure that essential needs are met, and where necessary, 
discretionary needs are met; 

WHEREAS, the areas affected include the counties of Macomb, Monroe, Oakland, 
Washtenaw, and Wayne; 

NOW, THEREFORE, I, JENNIFER M. GRANHOLM, Governor of the State of 
Michigan, pursuant to powers vested in me by the Michigan Constitution of 1963 and the 
provisions of the Emergency Management Act, 1976 PA 390, MCL 30.401 to 30.421, and 1982 
PA 191, MCL 10.81 to 10.87, proclaim: 

1. A state of emergency, including an energy emergency, exists in the counties of 
Macomb, Monroe, Oakland, Washtenaw, and Wayne. 

2. The response and recovery aspects of the Michigan Emergency Management Plan and 
the emergency operation plans of affected political subdivisions are activated to manage the state 
of emergency. 
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3. The Emergency Management Division of the Michigan Department of State Police 
(“EMD”) shall coordinate and maximize all state resources which may be activated to assist the 
affected areas in responding to the impact of the power outage and facilitate the restoration of 
power in the affected areas. The EMD may call upon all state departments to utilize resources at 
their disposal to assist in the emergency area pursuant to the Michigan Emergency Management 
Plan. 


4. Termination of this state of emergency will occur at such time as emergency 
conditions no longer exist and appropriate programs have been implemented to recover from the 
effects of this emergency, but in no event longer than September 14,2003, unless extended as 
provided by the Emergency Management Act, 1976 PA 390, MCL 30.401 to 30.421. 

This proclamation is effective immediately. 



WtED WITH S£C8ETAflT Of STATS 


Given under ray hand and the Great Seat of 
the State of Michigan this 15th day of 
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JENNIFER M. GRAN HOLM 
GOVERNOR 



Stats of Michigan 
OFFICE OF THE GOVERNOR 

LANSING 


Appendix A-3 


JOHN D, CHERRV. JR, 

IT. GOW-ftNOR 


EXECUTIVE ORDER 
2003 — 10 


TEMPORARY SUSPENSION OF ADMINISTRATIVE RULES FOR 
GASOLINE VAPOR PRESSURE 


WHEREAS, because significant portions of the State of Michigan have been experiencing 
the e ffects of a severe power outage, resulting in the loss of electrical power for countless Michigan 
residents, communities, and businesses, and causing serious hardship for the citizens of the State of 
Michigan, a state of emergency was declared by proclamation on August i 5,2003 in the counties of 
Macomb, Monroe, Oakland, Washtenaw, and Wayne (“State of Emergency”); 

WHEREAS, Section 5(l)(a) of the Emergency Management Act, 1976 PA 390, MCL 
30.405, empowers the Governor to suspend a regulatory statue, order, or rule prescribing the 
procedures for the conduct of state business when strict compliance with the statute, order, or rule 
would prevent, hinder, or delay necessary' action in coping with the disaster or emergency: 


NOW, THEREFORE, I, JENNIFER M. GRANIIOLM, Governor of the State of Michigan, 
pursuant to powers vested in me by the Michigan Constitution of 1963 and the provisions of the 
Emergency Management Act, 1976 PA 390, MCL 30.401 to 30.421, order: 


1. Administrative rules promulgated by the Department of Agriculture, Laboratory Division, 
dealing with gasoline vapor pressure, entitled, “Regulation No. 561 -Dispensing Facility Reid Vapor 
Pressure,” 1997 AACS, R 285.561.1 to 285.561.10, are suspended in (he areas of the State of 
Michigan subject to the State of Emergency and the counties of St. Clair and Livingston for the 
duration of the State of Emergency. 

This Order is effective immediately. 
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State 0* Michigan 
OFFICE OF THE GOVERNOR 

LANSING 


JOHN D. CHERRY, JR. 
IT. ClOvgANCft 


EXECUTIVE ORDER 
2003-11 

STATE OF ENERGY EMERGENCY 

WHEREAS, Article V, Section 1 of the Michigan Constitution of 1963 vests the 
executive power of the State of Michigan in the Governor; 

WHEREAS, Section 3 of 1 982 PA 191, MCL 10.83, authorizes the Governor to 
declare a State of Energy Emergency upon notification of an impending energy emergency 
by the Energy Advisoiy Committee, or upon the Governor's own initiative if the Governor 
finds that an energy emergency exists or is imminent; 

WHEREAS, on August 14,2003, a widespread and unprecedented loss of electrical 
power affected significant portions of the State of Michigan; 

WHEREAS, the power outage adversely impacted operations at eight petroleum 
refineries throughout the United States and Canada, and damaged Michigan's only refinery, 
which may be unable to meet demand for gasoline in the near future, resulting, without 
further action, in a lack of adequate available gasoline in parts of this state; 

WHEREAS, on August 20, 2003, the Public Service Commission notified the Energy 
Advisory Committee of an impending and imminent energy emergency involving a 
dwindling supply of gasoline in Southeast Michigan due to the power outage and damage to 
the refinery; 

WHEREAS, it is in the best interests of the State of Michigan that appropriate 
measures be taken in response t o an imminent energy emergency to ensure that gasoline 
supplies will remain sufficient and t o assure the health, safety, and welfare of Michigan 
residents and visitors; 
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NOW, THEREFORE, I, JENNIFER M. GRANHOLM, Governor of the State of 
Michigan, pursuant to powers vested in the Governor by the Michigan Constitution of 1963 
and 1982 PA 191, MCL 10.81 to 10.87, order the following: 

1. The State of Emergency proclaimed on August 15, 2003 for the counties of 
Macomb, Monroe, Oakland, Washtenaw, and Wayne is rescinded. 

2. A State of Energy Emergency is declared. Pursuant to Section 3 of 1982 PA 
191, MCL 10.83, the State of Energy Emergency is effective until the earlier of either of 
the following: 

a. A finding by the Governor that the energy emergency no longer exists 

b. November 19,2003. 



FILED WITH SECRET ART OF STATE 
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JENNIFER M. QRANHOUal 

GOViHMCR 


& 

State of Michigan 
OFFICE OF THE GOVERNOR 

LANSING 


( Appendix A-5 

! 


JOHN Q, CHERRY. JR. 

LT. COVSMnOH 


EXECUTIVE ORDER 
2003 ~ 12 

TEMPORARY SUSPENSION OF RULES FOR GASOLINE VAPOR PRESSURE 

WHEREAS, under 1982 PA 1981, MCL 10.83, during an energy emergency the 
Governor may by executive order suspend a rule of a state agency if strict compliance with the 
rule will prevent, hinder, or delay necessary action in coping with the emergency; 

WHEREAS, Executive Order 2003-1 1 declared a State of Energy Emergency beginning 
on August 21,2003; 

WHEREAS, appropriate measures must be taken in response to the energy emergency to 
ensure that gasoline supplies will remain sufficient and to assure the health, safety, and welfare 
of Michigan residents and visitors; 

NOW, THEREFORE, I, JENNIFER M. GRAN HOLM, Governor of the State of 
Michigan, pursuant to powers vested in die Governor by the Michigan Constitution of 1 963 and 
Michigan law, order that the Regulation No. 561, entitled, "Dispensing Facility Reid Vapor 
Pressure," promulgated by the Laboratory' Division of the Department of Agriculture, 1997 
AACS, R 285.561.1 to 285.561.10, be suspended for the duration of the energy emergency 
declared in Executive Order 2003-1 1. Additionally, Executive Order 2003-10 is rescinded. 

Given under my hand and the Great Seal 
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JENNIFER M, GRANHQtM 

GOVSBNGR 




State of Michigan 
OFFICE OF THE GOVERNOR 

LANSING 


Appendix A-6 


JOHN D. CHERRY. JR. 

tX. GOVERNOR 


EXECUTIVE ORDER 
2003-16 


END OF STATE OF ENERGY EMERGENCY 

WHEREAS, Article V, Section i of the Michigan Constitution of 1963 vests the 
executive power of the State of Michigan in the Governor; 

WHEREAS, under Section 3 of 1982 PA 191. MCL 10.83, a state of an energy 
emergency declared by the Governor is effective for the shorter of 90 days or until a finding 
that the energy emergency no longer exists; 

WHEREAS, the Chairperson of the Energy Advisory Committee has advised that the 
energy emergency recognized by Executive Order 2003-1 1 no longer exists; 

NOW, THEREFORE, I, JENNIFER M. GRANIiOLM, Governor of the Stale of 
Michigan, pursuant to powers vested in the Governor by die Michigan Constitution of 1963 
and Michigan law, order the following: 

1 . The state of energy emergency proclaimed on August 2 1 , 2003 under Executive Order 
2003-1 1 is rescinded, effective immediately. 

2. Executive Order 2003- 12 is rescinded, effective immediately. 



Given under my band and Ibe Great Seat 
of the State of Michigan this 30th day of 
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Subject; Portable Generators 


Contact: Margaret VauHaften 
(517)241.6165 
(mrvanha@mic higan. gov ) 
800.292.9555 


TIPS FOR BUYING AND USING A PORTABLE GENERATOR 

In the event of an electrical power outage, many Michigan homeowners ana businesses rely on 
portable power generators to keep lights and appliances running until service is restored. A 
portable generator is designed to run a limited number of appliances at a time and is typically 
powered by gasoline or diesel fuel. Generators usually cost between $600 and $3,000 ~ 
depending on size and features. The Michigan Public Service Commission (MPSC) wants you to 
consider some important points when deciding to buy and use a portable generator. 

Sizing 

To determine the size of the generator you will need, total the wattage of the lights and 
appliances you will need to power. For example : 


Appliance 

Wattage Needed to Run Appliance* 

Furnace (1/3 HP blower) 

1,200** 

Refrigerator 

600** 

Microwave oven 

700 

Two 100-watt light fixtures 

200 

■ 

Total 

2,700 


‘Appliance wattage varies — these figures represent averages. 

” Allow up to three times the normal running watts for starting these 
appliances or cycling their compressors. 


A typical portable generator is rated at 2,400 to 7,500 watts. Most household appliances are rated 
at 120 volts. Some larger electric appliances (e.g., electric range, electric clothes dryer, well 
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pump, air conditioner) are rated at 240 volts. If you want to power this type of appliance as well 
as smaller ones, you will need a generator that is rated at 120-240 volts. 

Installation 

Always read and follow all installation and operation instructions for your generator. There are 
two ways to safely install and operate a portable generator: 

Direct Hook-up 

Portable generators are designed to power a limited number of plug-in appliances like your 
refrigerator, freezer, and lights or any other combination of appliances you determine to be 
essential. These and other home appliances not permanently wired to the electrical system can be 
powered directly from the generator through a heavy-duty (at least 12 gauge), polarized 
extension cord. The extension cord should be less than 100 feet long to prevent power loss and 
overheating. 

Safety Transfer Switch 

Some generators can be permanently connected to your electric system to energize your home's 
wiring in the event of a power outage. This type of installation requires a safety transfer switch. 
Before starting your generator, you must activate the switch. The switch disconnects your home's 
wiring system from the electric company's system and allows electricity to flow from the 
generator to your home's circuitry. The switch prevents the generator from back- feeding 
electricity into the power lines and possibly causing injury or death to unsuspecting workers 
trying to restore power. The switch also prevents damage to your generator, wiring, and 
appliances when electric service is restored. Only a licensed electrician should install a transfer 
switch. 

Safety 


• Always follow the safety instructions in the manufacturer’s instruction manual. 

• Always follow local, state, and national fire and electric codes. A permit may be required 
for installation. 

• Always use a heavy-duty (at least 12 gauge) UL-Iisted extension cord (less than 100 feet 
long) from the generator to your appliances — being careful not to overload the cord. 

• Always make sure that the total electric load on your generator does not exceed the 
manufacturer's rating. 

• Always properly ground the generator according to the manufacturer's instructions. 

• Never operate a generator indoors or in an unventilated area. It produces deadly carbon 
monoxide fumes. 

• Always store gasoline and diesel fuel in approved containers and keep it out of the reach 
of children. 

• Never refuel a generator while it is running. Shut it off and let it cool for 10 minutes 
before refueling to minimize the danger of fire. 

• Parts of the generator are very hot during operation. Avoid contact and keep children 
away. 

• Protect the generator from rain and other moisture sources to prevent electrocution. 

• When not in use, store the generator in a dry location such as a garage or shed. 
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A portable generator can be a good, temporary source of electricity during a power outage. To 
avoid serious safety hazards when using a generator, it is important to follow all operation and 
safety instructions provided by the manufacturer. 

The Michigan Public Service Commission is an agency within the Department of Consumer and 
Industry Services. 

Alert 99-12 
September 30, 1999 


103 



480 


Appendix A-S 


SURVIVING ELECTRICAL POWER OUTAGES - WLIAT YOU CAN DO 
11* YOU LOSE YOUR ELECTRIC SERVICE 


Steviving 

Electrical 



Set aside and designate for emergency use: 

• Flashlight 

• Battery-powered radio 

• Extra batteries 

• Blankets 

• First-aid kit 

• Bottled water 

• Battery-operated lantern 

• Candles 


Keep a list of emergency numbers near the telephone. 

Protect electrical equipment such as a TV, VCR, microwave, or home computer with a voltage 
surge suppressor. A suppressor can eliminate the surge before it enters the equipment, thus 
protecting it from damage. A variety of devices are available for different forms of protection. If 
the equipment is not protected, unplug them before the storm begins to prevent lightning 
damage. 



• Check the fuse box to see if a fuse is blown or tripped. Check with the neighbors to see if 
their power is out. 

• Call your local utility company and let its personnel know that you have lost power. Also, 
advise if there is emergency medical equipment in the home. 

• Turn off and unplug most lights and appliances to prevent electrical overload when 
power is restored. 

• Keep refrigerator door closed as much as possible. Move milk, cheese, meats, etc. into 
the freezer compartment of the refrigerator. If the freezer is only partially lull, group 
packages together so they form an "igloo" to keep each other cold. Cover freezer with a 
blanket. Purchase dry ice and place in freezer. It will help keep food frozen for an 
extended period of time. 

• Make sure you have enough water for cooking and drinking. 

• Avoid downed power lines. 
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• If city water, open faucets so there is a constant drip so pipes won't freeze. 

• Hang cardboard or blankets over windows and doorways - find a well insulated room for 
living until power is restored. 

• Dress warmly - wear a hat because it helps prevent loss of body heat, since body heat 
escapes through the top of the head. 

• Fireplaces may be used to provide light as well as heat. Always keep the damper open for 
proper ventilation. 

• Store perishable food outside in a cold and shaded area or in an unheated garage. 



Wait a few minutes before turning on lights. Plug in appliances one at a time. 


105 



482 


Appendix A-9 

J. Peter Lark, Chair 

Robert B. Nelson, Commissioner 

Laura Chappelle, Commissioner 


Contact: Gary Kitts 517.241.6193 or 517.241-3323 www.micliigan.gov/mpsc 


Michigan Public Service Commission Urges Energy Conservation in Light of the Recent 
Blackouts 

August 15, 2003 

The Michigan Public Service Commission urges Michigan citizens to take all reasonable steps to 
conserve energy today in light of the devastating blackout through the Northeast. 

“Although power restoration is underway, the ability to complete and maintain that restoration 
will depend on the amount of demand on the system,” said Commission Chair J. Peter Lark. 

“We are asking all Michigan citizens to assist in this effort by conserving energy today to 
minimize the stress on the electric system. Almost 2 million Michigan customers remain without 
power this morning. Although most of the electric generating plants that went down are 
expected to be operating today, it may be several days before some of them are returned to 
service.” 

Here are some recommendations to help businesses and residential customers 
reduce your electric use: 

* If you're away from home for the day or your business is closed, turn off your central air 
conditioner or raise its setting above 78 degrees; 

* When home or at your business, raise the temperature on your central air conditioner to 
78 degrees or the highest setting comfortable; 

* Close off unoccupied areas and shut air-conditioning vents; 


(more) 
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Energy Conservation 

Page Two 

■ Close blinds, shades and drapes to keep your home or office more comfortable at higher 
temperature settings and help your fans and air conditioners work more efficiently; 

« Turn off all unnecessary lights, equipment and appliances; 

* Prepare meals that require little or no cooking; 

* Delay running your dishwasher, clothes washer and dryer until late evening; 

* Set fax machines and printers for sleep mode when not in use. Network one printer for 
several users; 

■ Make sure the power management feature is enabled on computers and set to the shortest 
acceptable time for your operation. Use laptops instead of personal computers. 

In most cases, regular telephone service will be available even though electric power is not. 

The MPSC is an agency within the Department of Consumer and Industry Services. 

### 
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George Voinovich 
Ohio 



UlCrrxfc^ -Slates -Senate 

WASHINGTON!, D. C. 20510 


September 2, 2003 

The Honorable Billy Tauzin, Chairman 
Committee on Energy and Commerce 
2125 Rayburn House Office Building 
United States House of Representatives 
Washington, DC 20515 

Dear Chairman Domenici & Chairman Tauzin: 


The Honorable Pete Domenici, Chairman 
Committee on Energy and Natural Resources 
364 Dirksen Senate Office Building 
United States Senate 
Washington, DC 20510 


As you are preparing to chair the House Senate Conference on H.R. 6, a comprehensive 
Energy Bill, I am writing to express my concerns about America’s electricity system and to 
request your support for legislation that will help strengthen that system. 

The recent blackout that affected parts of Ohio, as well as parts of Pennsylvania, New York, 
Michigan and Ontario, Canada, was only the most recent illustration of the vulnerability of 
our Nation’s electricity transmission system - a system that is in urgent need of 
modernization. Over the last 40 years, the system has become congested and strained 
because growth in electricity demand has not been matched by corresponding investment in 
new generation and transmission facilities. Further, our existing transmission infrastructure 
was not designed to meet present demand and daily transmission constraints increase the risk 
of blackouts. 


Ohio has long served as the center of our Nation’s manufacturing base, and currently has 
over 860,000 people in manufacturing jobs. Manufacturers in Ohio - and Nationwide - 
depend on adequate supplies of affordable and reliable electricity in order to remain 
competitive in the world market. 

In order to ensure that our manufacturers - as well as our health-care providers, educators, 
fanners and those working in the service, retail and technology sectors - have the energy 
they need to do their jobs, we must have an electricity system that includes both adequate 
generation and robust transmission. I respectfully urge the Conference Committee to 
consider factors that will encourage the construction of transmission and generation in places 
where it will be most beneficial to consumers. 

In order to encourage investment in the transmission grid, and to ensure the reliability of the 
grid, a comprehensive Energy Bill must include provisions that will: 

Ensure that the transmission system is reliable - Provisions to establish reliability 
standards to prevent instability and cascading failures of the system are included in 
both the House-passed and Senate-passed versions of H.R. 6. I encourage the 
conference to include such provisions in the Conference Report. 
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Protect regional deference - It is important that any language dealing with FERC’s 
Standard Market Design rulemaking contains an appropriate clarification that such 
language is not intended to interfere with regional efforts at market development and 
reliability improvements underway by established Regional Transmission 
Organizations. It is also important that any provisions included in the Energy Bill 
will not interfere with regional efforts to address issues associated with such market 
development and reliability improvement efforts. 

Facilitate siting of new transmission corridors - Last year I sponsored legislation 
(S. 1590) to amend the National Environmental Policy Act of 1969 and streamline the 
siting process for transmission corridors. I encourage the conference to adopt this 
language, which I have attached, as part of the Energy Bill. 

Further, in order to ensure, that we have an adequate supply of electricity, the Energy Bill 
must include provisions that will: 

Increase domestic production of natural gas - Both the House-passed and Senate- 
passed version of H.R. 6 include provisions to increase domestic production of 
natural gas and to ensure that we have a healthy, vital fuel mix for electric generation. 
These are important provisions that should be included in the Conference Report. 

Promote alternatives to natural gas - Both the House-passed and Senate-passed 
version of H.R. 6 include provisions to encourage alternatives to natural gas for 
electricity generation - such as nuclear power, hydropower, renewable energy and 
clean coal - which are vital in order to relieve the pressure on our natural gas supply. 
Continued and expanded use of these fuels to generate electricity is vital to having a 
healthy generation mix. 

Finally, efforts to regulate emissions of carbon will force our utilities to fuel switch to natural 
gas, will significantly raise energy prices, will cause thousands of jobs to be lost and are 
directly contrary to the policies that we are trying advance in passing a comprehensive 
Energy Bill. The conferees should not include any language on climate change in the 
Conference Report. 

Encouraging the development of robust and reliable transmission, as well as adequate 
supplies of electricity, will help to ensure that we do not face blackouts such as the one that 
recently hit our Nation. Thank you for your consideration of my concerns. 


Sincerely, 



United States Senator 
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PAT WOOD, III, RESPONSES TO QUESTIONS 
FROM SENATOR FRANK R. LAUTENBERG 

Question 1 : Mr. Wood, the newspapers have reported on a deal the 
Administration made to delay FERC’s Standard Market Design Rule, and prevent the 
Commission from requiring mandatory participation in RTOs until 2007. 

If electricity deregulation and RTO participation are dramatically slowed, what 
effect do you think that will have on the risks of future blackouts? 

Answer : At this time, we are still working with the United States - Canada 

Joint Task Force to determine what caused the August 14, 2003 blackout that cascaded 
through eight states and parts of Canada. 

As to the risk of future blackouts, however, one of the ways in which the 
Commission is fostering a more reliable electricity system is by promoting regional 
coordination and planning of the interstate grid through regional transmission 
organizations (RTOs) and independent system operators (ISOs). (The major distinction 
between the two entities is that an ISO can be geographically smaller than an RTO). 

They can also bolster the reliability of the grid through unified operation of the 
transmission system across a broad region. Participation in RTOs and ISOs provide these 
benefits to both regulated and competitive retail markets. 

In Order No. 2000, the Commission recognized that regional organizations have 
unique advantages to assist in regional planning for new transmission infrastructure. The 
Commission required that RTOs have a regional planning process to identify and arrange 
for necessary transmission additions and upgrades. With respect to operating the 
interstate transmission grid, in Order No. 2000, the Commission identified the benefits of 
large, independent regional entities to operate the grid, and strongly encouraged, but did 
not require, utilities to join together to form such entities. The Commission noted that 
such entities would improve reliability because they have a broader, more regional 
perspective on electrical operations than many separate utilities. In addition, some 130 
control area operators currently manage the operation of the transmission grid, whereas a 
smaller number of regional organizations could manage the grid more effectively. 

Further, unlike utilities that own both generation and transmission, RTOs are independent 
of market participants and, therefore, lack a financial incentive to use the transmission 
grid to benefit any one market participant. 
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Question 2 : Mr. Wood, yesterday I was visited by several employees of 
FirstEnergy. They reported to me that FirstEnergy is using power transmission lines at 
well-over capacity and has failed to replace sections of old, unreliable infrastructure. Can 
FERC play a role to address this sort of dangerous and irresponsible behavior on the part 
of a utility? 

Answer : The Commission has no explicit authority to inspect or require the 

replacement of old, unreliable infrastructure. Historically, state regulators carried out this 
function. However, the Commission has offered an enhanced return on equity for new 
transmission infrastructure in rates under its jurisdiction. More generally, there is no 
direct federal authority or responsibility for the reliability of the transmission grid. The 
Federal Power Act contains only limited authorities on reliability. Congress is now 
considering, in conference, energy legislation that provides for mandatory electric 
reliability rules, subject to Commission oversight. The mandatory electric reliability 
standards authorized in the Senate version of H.R. 6 would allow the Commission to 
oversee short-term reliability by approving requirements for the operation of the bulk- 
power system. My understanding is that these requirements could address, and limit, 
intentional overloading of transmission lines. 

I believe that mandatory reliability standards are critical to our Nation's security. 

If Congress wishes to establish an even stronger federal role in ensuring long-term 
reliability of the bulk-power system, H.R. 6 could be revised to authorize an electric 
reliability organization, subject to Commission oversight, to order the construction of 
additional transmission capacity. 


o 



